Sep 09 20:50:17.368138 ip-10-0-138-224 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 09 20:50:17.368154 ip-10-0-138-224 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 09 20:50:17.368163 ip-10-0-138-224 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 09 20:50:17.368365 ip-10-0-138-224 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 09 20:50:27.492994 ip-10-0-138-224 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 09 20:50:27.493012 ip-10-0-138-224 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot caab0152198040d69100d2feb5a39922 --
Sep 09 20:52:22.667739 ip-10-0-138-224 systemd[1]: Starting Kubernetes Kubelet...
Sep 09 20:52:23.059039 ip-10-0-138-224 kubenswrapper[2579]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 09 20:52:23.059039 ip-10-0-138-224 kubenswrapper[2579]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 09 20:52:23.059039 ip-10-0-138-224 kubenswrapper[2579]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 09 20:52:23.059039 ip-10-0-138-224 kubenswrapper[2579]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 09 20:52:23.059039 ip-10-0-138-224 kubenswrapper[2579]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 09 20:52:23.061062 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.060979    2579 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 09 20:52:23.065271 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.065257    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 20:52:23.065271 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.065271    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 20:52:23.065336 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.065275    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 20:52:23.065336 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.065278    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 20:52:23.065336 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.065281    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 20:52:23.065336 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.065284    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 20:52:23.065336 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.065286    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 20:52:23.065336 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.065290    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 20:52:23.065336 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.065293    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 20:52:23.065336 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.065296    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 20:52:23.065336 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.065299    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 20:52:23.065336 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.065301    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 20:52:23.065336 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.065304    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 20:52:23.065336 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.065307    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 20:52:23.065336 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.065311    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 20:52:23.065336 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.065316    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 20:52:23.065336 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.065318    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 20:52:23.065336 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.065322    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 20:52:23.065336 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.065325    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 20:52:23.065336 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.065327    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 20:52:23.065336 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.065330    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 20:52:23.065336 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.065333    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 20:52:23.066607 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.065336    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 20:52:23.066607 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.065338    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 20:52:23.066607 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.065341    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 20:52:23.066607 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.065344    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 20:52:23.066607 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.065347    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 20:52:23.066607 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.065349    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 20:52:23.066607 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.065352    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 20:52:23.066607 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.065355    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 20:52:23.066607 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.065358    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 20:52:23.066607 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.065360    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 20:52:23.066607 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.065363    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 20:52:23.066607 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.065365    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 20:52:23.066607 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.065368    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 20:52:23.066607 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.065370    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 20:52:23.066607 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.065373    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 20:52:23.066607 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.065375    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 20:52:23.066607 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.065379    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 20:52:23.066607 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.065382    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 20:52:23.066607 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.065384    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 20:52:23.067076 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.065388    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 20:52:23.067076 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.065391    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 20:52:23.067076 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.065394    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 20:52:23.067076 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.065397    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 20:52:23.067076 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.065400    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 20:52:23.067076 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.065402    2579 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 20:52:23.067076 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.065405    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 20:52:23.067076 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.065407    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 20:52:23.067076 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.065410    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 20:52:23.067076 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.065412    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 20:52:23.067076 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.065415    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 20:52:23.067076 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.065417    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 20:52:23.067076 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.065420    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 20:52:23.067076 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.065423    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 20:52:23.067076 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.065427    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 20:52:23.067076 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.065429    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 20:52:23.067076 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.065432    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 20:52:23.067076 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.065434    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 20:52:23.067076 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.065450    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 20:52:23.067076 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.065452    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 20:52:23.067598 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.065455    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 20:52:23.067598 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.065458    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 20:52:23.067598 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.065461    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 20:52:23.067598 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.065463    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 20:52:23.067598 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.065466    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 20:52:23.067598 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.065468    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 20:52:23.067598 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.065471    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 20:52:23.067598 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.065475    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 20:52:23.067598 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.065478    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 20:52:23.067598 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.065481    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 20:52:23.067598 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.065484    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 20:52:23.067598 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.065487    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 20:52:23.067598 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.065489    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 20:52:23.067598 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.065492    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 20:52:23.067598 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.065494    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 20:52:23.067598 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.065497    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 20:52:23.067598 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.065500    2579 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 20:52:23.067598 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.065503    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 20:52:23.067598 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.065505    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 20:52:23.068185 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.065507    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 20:52:23.068185 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.065510    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 20:52:23.068185 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.065513    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 20:52:23.068185 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.065515    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 20:52:23.068185 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.065518    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 20:52:23.068185 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.065521    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 20:52:23.068185 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.065523    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 20:52:23.068185 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.065526    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 20:52:23.068185 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.065529    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 20:52:23.068185 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.065531    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 20:52:23.068185 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.065534    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 20:52:23.068185 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.065536    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 20:52:23.068185 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.065539    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 20:52:23.068185 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.065541    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 20:52:23.068185 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.065543    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 20:52:23.068185 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.065546    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 20:52:23.068185 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.065549    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 20:52:23.068185 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.065551    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 20:52:23.068185 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.065554    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 20:52:23.068660 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.066269    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 20:52:23.068660 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.066276    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 20:52:23.068660 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.066279    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 20:52:23.068660 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.066283    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 20:52:23.068660 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.066301    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 20:52:23.068660 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.066306    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 20:52:23.068660 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.066309    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 20:52:23.068660 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.066312    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 20:52:23.068660 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.066315    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 20:52:23.068660 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.066318    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 20:52:23.068660 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.066321    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 20:52:23.068660 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.066323    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 20:52:23.068660 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.066325    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 20:52:23.068660 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.066328    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 20:52:23.068660 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.066331    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 20:52:23.068660 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.066333    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 20:52:23.068660 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.066336    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 20:52:23.068660 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.066338    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 20:52:23.068660 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.066341    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 20:52:23.068660 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.066344    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 20:52:23.069158 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.066346    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 20:52:23.069158 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.066350    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 20:52:23.069158 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.066354    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 20:52:23.069158 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.066357    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 20:52:23.069158 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.066359    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 20:52:23.069158 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.066362    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 20:52:23.069158 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.066365    2579 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 20:52:23.069158 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.066367    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 20:52:23.069158 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.066370    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 20:52:23.069158 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.066373    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 20:52:23.069158 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.066375    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 20:52:23.069158 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.066377    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 20:52:23.069158 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.066380    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 20:52:23.069158 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.066382    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 20:52:23.069158 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.066385    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 20:52:23.069158 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.066387    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 20:52:23.069158 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.066390    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 20:52:23.069158 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.066392    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 20:52:23.069158 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.066395    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 20:52:23.069158 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.066397    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 20:52:23.069668 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.066400    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 20:52:23.069668 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.066402    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 20:52:23.069668 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.066405    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 20:52:23.069668 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.066407    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 20:52:23.069668 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.066409    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 20:52:23.069668 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.066412    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 20:52:23.069668 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.066414    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 20:52:23.069668 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.066417    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 20:52:23.069668 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.066421    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 20:52:23.069668 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.066423    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 20:52:23.069668 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.066427    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 20:52:23.069668 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.066430    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 20:52:23.069668 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.066452    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 20:52:23.069668 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.066455    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 20:52:23.069668 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.066457    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 20:52:23.069668 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.066460    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 20:52:23.069668 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.066463    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 20:52:23.069668 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.066465    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 20:52:23.069668 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.066468    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 20:52:23.070119 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.066470    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 20:52:23.070119 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.066473    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 20:52:23.070119 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.066475    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 20:52:23.070119 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.066478    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 20:52:23.070119 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.066480    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 20:52:23.070119 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.066483    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 20:52:23.070119 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.066485    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 20:52:23.070119 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.066488    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 20:52:23.070119 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.066490    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 20:52:23.070119 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.066493    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 20:52:23.070119 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.066496    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 20:52:23.070119 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.066500    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 20:52:23.070119 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.066503    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 20:52:23.070119 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.066507    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 20:52:23.070119 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.066510    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 20:52:23.070119 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.066514    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 20:52:23.070119 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.066518    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 20:52:23.070119 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.066524    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 20:52:23.070119 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.066529    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 20:52:23.070591 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.066546    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 20:52:23.070591 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.066550    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 20:52:23.070591 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.066552    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 20:52:23.070591 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.066555    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 20:52:23.070591 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.066558    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 20:52:23.070591 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.066560    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 20:52:23.070591 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.066565    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 20:52:23.070591 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.066567    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 20:52:23.070591 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.066570    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 20:52:23.070591 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.066572    2579 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 20:52:23.070591 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.066575    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 20:52:23.070591 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.066577    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 20:52:23.070591 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.066580    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 20:52:23.070591 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.066582    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 20:52:23.070591 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.066585    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 20:52:23.070591 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.066587    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 20:52:23.070591 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.066590    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 20:52:23.070591 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.066592    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 20:52:23.070591 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.066595    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 20:52:23.070591 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.066597    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 20:52:23.071069 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.066600    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 20:52:23.071069 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.066958    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 20:52:23.071069 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.066963    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 20:52:23.071069 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.066966    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 20:52:23.071069 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.066968    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 20:52:23.071069 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.066972    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 20:52:23.071069 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.066976    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 20:52:23.071069 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.066979    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 20:52:23.071069 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.066982    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 20:52:23.071069 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.066984    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 20:52:23.071069 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.066987    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 20:52:23.071069 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.066990    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 20:52:23.071069 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.066992    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 20:52:23.071069 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.066996    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 20:52:23.071069 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.067000    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 20:52:23.071069 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.067003    2579 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 20:52:23.071069 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.067005    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 20:52:23.071069 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.067008    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 20:52:23.071069 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.067012    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 20:52:23.071069 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.067015    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 20:52:23.071570 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.067018    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 20:52:23.071570 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.067020    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 20:52:23.071570 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.067023    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 20:52:23.071570 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.067026    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 20:52:23.071570 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.067028    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 20:52:23.071570 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.067030    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 20:52:23.071570 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.067033    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 20:52:23.071570 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.067035    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 20:52:23.071570 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.067038    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 20:52:23.071570 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.067040    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 20:52:23.071570 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.067043    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 20:52:23.071570 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.067045    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 20:52:23.071570 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.067048    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 20:52:23.071570 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.067050    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 20:52:23.071570 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.067053    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 20:52:23.071570 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.067055    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 20:52:23.071570 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.067058    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 20:52:23.071570 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.067060    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 20:52:23.071570 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.067066    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 20:52:23.072027 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.067069    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 20:52:23.072027 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.067072    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 20:52:23.072027 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.067075    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 20:52:23.072027 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.067078    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 20:52:23.072027 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.067081    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 20:52:23.072027 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.067084    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 20:52:23.072027 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.067087    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 20:52:23.072027 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.067090    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 20:52:23.072027 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.067092    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 20:52:23.072027 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.067095    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 20:52:23.072027 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.067097    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 20:52:23.072027 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.067101    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 20:52:23.072027 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.067104    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 20:52:23.072027 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.067106    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 20:52:23.072027 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.067109    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 20:52:23.072027 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.067111    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 20:52:23.072027 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.067113    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 20:52:23.072027 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.067116    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 20:52:23.072027 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.067118    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 20:52:23.072027 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.067121    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 20:52:23.072530 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.067123    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 20:52:23.072530 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.067126    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 20:52:23.072530 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.067128    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 20:52:23.072530 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.067131    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 20:52:23.072530 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.067134    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 20:52:23.072530 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.067136    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 20:52:23.072530 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.067139    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 20:52:23.072530 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.067141    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 20:52:23.072530 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.067144    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 20:52:23.072530 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.067146    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 20:52:23.072530 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.067149    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 20:52:23.072530 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.067152    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 20:52:23.072530 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.067155    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 20:52:23.072530 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.067157    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 20:52:23.072530 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.067160    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 20:52:23.072530 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.067163    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 20:52:23.072530 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.067166    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 20:52:23.072530 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.067169    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 20:52:23.072530 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.067171    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 20:52:23.073026 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.067174    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 20:52:23.073026 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.067176    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 20:52:23.073026 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.067178    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 20:52:23.073026 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.067181    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 20:52:23.073026 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.067185    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 20:52:23.073026 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.067187    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 20:52:23.073026 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.067189    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 20:52:23.073026 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.067192    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 20:52:23.073026 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.067194    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 20:52:23.073026 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.067197    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 20:52:23.073026 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.067199    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 20:52:23.073026 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.067202    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 20:52:23.073026 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.067204    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 20:52:23.073026 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.067207    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 20:52:23.073026 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.067209    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 20:52:23.073026 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.067212    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 20:52:23.073026 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.067214    2579 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 20:52:23.073026 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.067216    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 20:52:23.073026 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.067219    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 20:52:23.073026 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.067221    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 20:52:23.073532 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.067224    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 20:52:23.073532 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.067226    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 20:52:23.073532 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.067300    2579 flags.go:64] FLAG: --address="0.0.0.0"
Sep 09 20:52:23.073532 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.067333    2579 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 09 20:52:23.073532 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.067339    2579 flags.go:64] FLAG: --anonymous-auth="true"
Sep 09 20:52:23.073532 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.067343    2579 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 09 20:52:23.073532 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.067347    2579 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 09 20:52:23.073532 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.067351    2579 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 09 20:52:23.073532 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.067355    2579 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 09 20:52:23.073532 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.067358    2579 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 09 20:52:23.073532 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.067361    2579 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 09 20:52:23.073532 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.067364    2579 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 09 20:52:23.073532 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.067368    2579 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 09 20:52:23.073532 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.067371    2579 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 09 20:52:23.073532 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.067374    2579 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 09 20:52:23.073532 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.067376    2579 flags.go:64] FLAG: --cgroup-root=""
Sep 09 20:52:23.073532 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.067380    2579 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 09 20:52:23.073532 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.067384    2579 flags.go:64] FLAG: --client-ca-file=""
Sep 09 20:52:23.073532 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.067386    2579 flags.go:64] FLAG: --cloud-provider="external"
Sep 09 20:52:23.073532 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.067389    2579 flags.go:64] FLAG: --cluster-dns="[]"
Sep 09 20:52:23.073532 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.067394    2579 flags.go:64] FLAG: --cluster-domain=""
Sep 09 20:52:23.073532 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.067396    2579 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 09 20:52:23.073532 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.067399    2579 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 09 20:52:23.074085 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.067402    2579 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 09 20:52:23.074085 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.067405    2579 flags.go:64] FLAG: --container-log-max-files="5"
Sep 09 20:52:23.074085 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.067409    2579 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 09 20:52:23.074085 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.067412    2579 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 09 20:52:23.074085 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.067415    2579 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 09 20:52:23.074085 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.067418    2579 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 09 20:52:23.074085 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.067421    2579 flags.go:64] FLAG: --contention-profiling="false"
Sep 09 20:52:23.074085 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.067423    2579 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 09 20:52:23.074085 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.067426    2579 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 09 20:52:23.074085 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.067429    2579 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 09 20:52:23.074085 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.067432    2579 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 09 20:52:23.074085 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.067453    2579 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 09 20:52:23.074085 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.067458    2579 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 09 20:52:23.074085 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.067462    2579 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 09 20:52:23.074085 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.067467    2579 flags.go:64] FLAG: --enable-load-reader="false"
Sep 09 20:52:23.074085 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.067470    2579 flags.go:64] FLAG: --enable-server="true"
Sep 09 20:52:23.074085 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.067474    2579 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 09 20:52:23.074085 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.067478    2579 flags.go:64] FLAG: --event-burst="100"
Sep 09 20:52:23.074085 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.067481    2579 flags.go:64] FLAG: --event-qps="50"
Sep 09 20:52:23.074085 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.067484    2579 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 09 20:52:23.074085 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.067487    2579 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 09 20:52:23.074085 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.067490    2579 flags.go:64] FLAG: --eviction-hard=""
Sep 09 20:52:23.074085 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.067493    2579 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 09 20:52:23.074085 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.067496    2579 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 09 20:52:23.074658 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.067499    2579 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 09 20:52:23.074658 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.067503    2579 flags.go:64] FLAG: --eviction-soft=""
Sep 09 20:52:23.074658 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.067506    2579 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 09 20:52:23.074658 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.067509    2579 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 09 20:52:23.074658 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.067511    2579 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 09 20:52:23.074658 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.067514    2579 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 09 20:52:23.074658 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.067517    2579 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 09 20:52:23.074658 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.067521    2579 flags.go:64] FLAG: --fail-swap-on="true"
Sep 09 20:52:23.074658 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.067524    2579 flags.go:64] FLAG: --feature-gates=""
Sep 09 20:52:23.074658 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.067530    2579 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 09 20:52:23.074658 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.067534    2579 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 09 20:52:23.074658 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.067537    2579 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 09 20:52:23.074658 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.067540    2579 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 09 20:52:23.074658 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.067543    2579 flags.go:64] FLAG: --healthz-port="10248"
Sep 09 20:52:23.074658 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.067545    2579 flags.go:64] FLAG: --help="false"
Sep 09 20:52:23.074658 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.067548    2579 flags.go:64] FLAG: --hostname-override="ip-10-0-138-224.ec2.internal"
Sep 09 20:52:23.074658 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.067551    2579 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 09 20:52:23.074658 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.067554    2579 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 09 20:52:23.074658 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.067557    2579 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 09 20:52:23.074658 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.067560    2579 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 09 20:52:23.074658 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.067564    2579 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 09 20:52:23.074658 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.067567    2579 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 09 20:52:23.074658 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.067569    2579 flags.go:64] FLAG: --image-service-endpoint=""
Sep 09 20:52:23.075224 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.067572    2579 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 09 20:52:23.075224 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.067575    2579 flags.go:64] FLAG: --kube-api-burst="100"
Sep 09 20:52:23.075224 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.067578    2579 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 09 20:52:23.075224 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.067581    2579 flags.go:64] FLAG: --kube-api-qps="50"
Sep 09 20:52:23.075224 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.067584    2579 flags.go:64] FLAG: --kube-reserved=""
Sep 09 20:52:23.075224 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.067586    2579 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 09 20:52:23.075224 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.067589    2579 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 09 20:52:23.075224 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.067592    2579 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 09 20:52:23.075224 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.067595    2579 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 09 20:52:23.075224 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.067598    2579 flags.go:64] FLAG: --lock-file=""
Sep 09 20:52:23.075224 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.067602    2579 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 09 20:52:23.075224 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.067605    2579 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 09 20:52:23.075224 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.067608    2579 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 09 20:52:23.075224 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.067613    2579 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 09 20:52:23.075224 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.067616    2579 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 09 20:52:23.075224 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.067618    2579 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 09 20:52:23.075224 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.067621    2579 flags.go:64] FLAG: --logging-format="text"
Sep 09 20:52:23.075224 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.067624    2579 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 09 20:52:23.075224 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.067627    2579 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 09 20:52:23.075224 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.067630    2579 flags.go:64] FLAG: --manifest-url=""
Sep 09 20:52:23.075224 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.067633    2579 flags.go:64] FLAG: --manifest-url-header=""
Sep 09 20:52:23.075224 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.067637    2579 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 09 20:52:23.075224 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.067640    2579 flags.go:64] FLAG: --max-open-files="1000000"
Sep 09 20:52:23.075224 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.067643    2579 flags.go:64] FLAG: --max-pods="110"
Sep 09 20:52:23.075224 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.067646    2579 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 09 20:52:23.075875 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.067649    2579 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 09 20:52:23.075875 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.067652    2579 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 09 20:52:23.075875 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.067655    2579 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 09 20:52:23.075875 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.067662    2579 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 09 20:52:23.075875 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.067665    2579 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 09 20:52:23.075875 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.067668    2579 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 09 20:52:23.075875 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.067675    2579 flags.go:64] FLAG: --node-status-max-images="50"
Sep 09 20:52:23.075875 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.067678    2579 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 09 20:52:23.075875 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.067686    2579 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 09 20:52:23.075875 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.067689    2579 flags.go:64] FLAG: --pod-cidr=""
Sep 09 20:52:23.075875 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.067693    2579 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ccbacc999dbf9abad8f4009240188312b683f36ced4763eed166c1500c164b1"
Sep 09 20:52:23.075875 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.067698    2579 flags.go:64] FLAG: --pod-manifest-path=""
Sep 09 20:52:23.075875 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.067701    2579 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 09 20:52:23.075875 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.067704    2579 flags.go:64] FLAG: --pods-per-core="0"
Sep 09 20:52:23.075875 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.067707    2579 flags.go:64] FLAG: --port="10250"
Sep 09 20:52:23.075875 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.067710    2579 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 09 20:52:23.075875 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.067713    2579 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-03950702f1b931bed"
Sep 09 20:52:23.075875 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.067716    2579 flags.go:64] FLAG: --qos-reserved=""
Sep 09 20:52:23.075875 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.067720    2579 flags.go:64] FLAG: --read-only-port="10255"
Sep 09 20:52:23.075875 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.067723    2579 flags.go:64] FLAG: --register-node="true"
Sep 09 20:52:23.075875 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.067726    2579 flags.go:64] FLAG: --register-with-taints=""
Sep 09 20:52:23.075875 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.067729    2579 flags.go:64] FLAG: --registry-burst="10"
Sep 09 20:52:23.075875 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.067732    2579 flags.go:64] FLAG: --registry-qps="5"
Sep 09 20:52:23.075875 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.067734    2579 flags.go:64] FLAG: --reserved-cpus=""
Sep 09 20:52:23.076741 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.067737    2579 flags.go:64] FLAG: --reserved-memory=""
Sep 09 20:52:23.076741 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.067741    2579 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 09 20:52:23.076741 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.067744    2579 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 09 20:52:23.076741 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.067746    2579 flags.go:64] FLAG: --rotate-certificates="false"
Sep 09 20:52:23.076741 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.067749    2579 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 09 20:52:23.076741 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.067752    2579 flags.go:64] FLAG: --runonce="false"
Sep 09 20:52:23.076741 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.067755    2579 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 09 20:52:23.076741 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.067758    2579 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 09 20:52:23.076741 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.067761    2579 flags.go:64] FLAG: --seccomp-default="false"
Sep 09 20:52:23.076741 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.067763    2579 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 09 20:52:23.076741 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.067766    2579 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 09 20:52:23.076741 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.067771    2579 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 09 20:52:23.076741 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.067773    2579 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 09 20:52:23.076741 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.067776    2579 flags.go:64] FLAG: --storage-driver-password="root"
Sep 09 20:52:23.076741 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.067779    2579 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 09 20:52:23.076741 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.067782    2579 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 09 20:52:23.076741 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.067785    2579 flags.go:64] FLAG: --storage-driver-user="root"
Sep 09 20:52:23.076741 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.067787    2579 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 09 20:52:23.076741 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.067790    2579 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 09 20:52:23.076741 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.067794    2579 flags.go:64] FLAG: --system-cgroups=""
Sep 09 20:52:23.076741 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.067796    2579 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 09 20:52:23.076741 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.067801    2579 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 09 20:52:23.076741 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.067804    2579 flags.go:64] FLAG: --tls-cert-file=""
Sep 09 20:52:23.076741 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.067807    2579 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 09 20:52:23.076741 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.067811    2579 flags.go:64] FLAG: --tls-min-version=""
Sep 09 20:52:23.077555 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.067814    2579 flags.go:64] FLAG: --tls-private-key-file=""
Sep 09 20:52:23.077555 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.067818    2579 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 09 20:52:23.077555 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.067820    2579 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 09 20:52:23.077555 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.067823    2579 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 09 20:52:23.077555 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.067826    2579 flags.go:64] FLAG: --v="2"
Sep 09 20:52:23.077555 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.067830    2579 flags.go:64] FLAG: --version="false"
Sep 09 20:52:23.077555 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.067834    2579 flags.go:64] FLAG: --vmodule=""
Sep 09 20:52:23.077555 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.067837    2579 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 09 20:52:23.077555 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.067840    2579 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 09 20:52:23.077555 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.067950    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 20:52:23.077555 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.067954    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 20:52:23.077555 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.067957    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 20:52:23.077555 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.067960    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 20:52:23.077555 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.067963    2579 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 20:52:23.077555 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.067966    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 20:52:23.077555 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.067969    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 20:52:23.077555 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.067972    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 20:52:23.077555 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.067975    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 20:52:23.077555 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.067979    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 20:52:23.077555 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.067983    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 20:52:23.077555 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.067987    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 20:52:23.078062 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.067990    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 20:52:23.078062 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.067992    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 20:52:23.078062 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.067995    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 20:52:23.078062 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.067998    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 20:52:23.078062 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.068001    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 20:52:23.078062 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.068004    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 20:52:23.078062 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.068007    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 20:52:23.078062 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.068010    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 20:52:23.078062 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.068012    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 20:52:23.078062 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.068015    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 20:52:23.078062 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.068018    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 20:52:23.078062 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.068020    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 20:52:23.078062 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.068024    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 20:52:23.078062 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.068027    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 20:52:23.078062 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.068029    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 20:52:23.078062 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.068032    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 20:52:23.078062 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.068034    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 20:52:23.078062 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.068037    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 20:52:23.078062 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.068039    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 20:52:23.078062 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.068042    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 20:52:23.078571 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.068044    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 20:52:23.078571 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.068062    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 20:52:23.078571 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.068067    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 20:52:23.078571 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.068070    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 20:52:23.078571 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.068073    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 20:52:23.078571 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.068075    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 20:52:23.078571 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.068079    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 20:52:23.078571 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.068081    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 20:52:23.078571 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.068084    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 20:52:23.078571 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.068088    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 20:52:23.078571 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.068091    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 20:52:23.078571 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.068093    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 20:52:23.078571 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.068096    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 20:52:23.078571 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.068099    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 20:52:23.078571 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.068101    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 20:52:23.078571 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.068104    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 20:52:23.078571 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.068106    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 20:52:23.078571 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.068109    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 20:52:23.078571 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.068111    2579 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 20:52:23.079032 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.068114    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 20:52:23.079032 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.068117    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 20:52:23.079032 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.068119    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 20:52:23.079032 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.068121    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 20:52:23.079032 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.068124    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 20:52:23.079032 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.068127    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 20:52:23.079032 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.068130    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 20:52:23.079032 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.068132    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 20:52:23.079032 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.068135    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 20:52:23.079032 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.068137    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 20:52:23.079032 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.068140    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 20:52:23.079032 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.068142    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 20:52:23.079032 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.068145    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 20:52:23.079032 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.068147    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 20:52:23.079032 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.068150    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 20:52:23.079032 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.068152    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 20:52:23.079032 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.068155    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 20:52:23.079032 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.068157    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 20:52:23.079032 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.068160    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 20:52:23.079032 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.068162    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 20:52:23.079544 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.068165    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 20:52:23.079544 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.068167    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 20:52:23.079544 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.068170    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 20:52:23.079544 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.068173    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 20:52:23.079544 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.068176    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 20:52:23.079544 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.068178    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 20:52:23.079544 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.068181    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 20:52:23.079544 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.068183    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 20:52:23.079544 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.068186    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 20:52:23.079544 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.068188    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 20:52:23.079544 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.068191    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 20:52:23.079544 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.068193    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 20:52:23.079544 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.068197    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 20:52:23.079544 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.068201    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 20:52:23.079544 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.068207    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 20:52:23.079544 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.068212    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 20:52:23.079544 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.068215    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 20:52:23.079544 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.068219    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 20:52:23.079544 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.068222    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 20:52:23.079544 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.068225    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 20:52:23.080061 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.068227    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 20:52:23.080061 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.068230    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 20:52:23.080061 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.068232    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 20:52:23.080061 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.068235    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 20:52:23.080061 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.068238    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 20:52:23.080061 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.068240    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 20:52:23.080061 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.068242    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 20:52:23.080061 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.068245    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 20:52:23.080061 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.069119    2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 09 20:52:23.080061 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.078476    2579 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 09 20:52:23.080061 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.078490    2579 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 09 20:52:23.080061 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.078509    2579 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 09 20:52:23.080061 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.078516    2579 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 09 20:52:23.080061 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.078573    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 20:52:23.080061 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.078577    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 20:52:23.080451 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.078581    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 20:52:23.080451 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.078584    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 20:52:23.080451 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.078587    2579 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 20:52:23.080451 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.078590    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 20:52:23.080451 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.078593    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 20:52:23.080451 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.078596    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 20:52:23.080451 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.078599    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 20:52:23.080451 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.078601    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 20:52:23.080451 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.078604    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 20:52:23.080451 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.078607    2579 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 20:52:23.080451 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.078610    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 20:52:23.080451 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.078613    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 20:52:23.080451 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.078615    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 20:52:23.080451 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.078618    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 20:52:23.080451 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.078620    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 20:52:23.080451 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.078623    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 20:52:23.080451 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.078625    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 20:52:23.080451 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.078628    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 20:52:23.080451 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.078630    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 20:52:23.080451 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.078632    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 20:52:23.080451 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.078635    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 20:52:23.080963 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.078637    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 20:52:23.080963 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.078640    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 20:52:23.080963 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.078648    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 20:52:23.080963 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.078652    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 20:52:23.080963 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.078656    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 20:52:23.080963 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.078659    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 20:52:23.080963 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.078663    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 20:52:23.080963 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.078666    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 20:52:23.080963 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.078669    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 20:52:23.080963 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.078672    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 20:52:23.080963 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.078674    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 20:52:23.080963 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.078677    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 20:52:23.080963 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.078679    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 20:52:23.080963 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.078682    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 20:52:23.080963 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.078685    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 20:52:23.080963 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.078687    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 20:52:23.080963 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.078690    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 20:52:23.080963 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.078692    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 20:52:23.080963 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.078695    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 20:52:23.081421 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.078697    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 20:52:23.081421 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.078701    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 20:52:23.081421 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.078705    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 20:52:23.081421 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.078708    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 20:52:23.081421 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.078710    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 20:52:23.081421 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.078713    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 20:52:23.081421 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.078715    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 20:52:23.081421 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.078718    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 20:52:23.081421 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.078720    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 20:52:23.081421 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.078723    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 20:52:23.081421 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.078726    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 20:52:23.081421 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.078728    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 20:52:23.081421 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.078731    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 20:52:23.081421 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.078733    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 20:52:23.081421 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.078736    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 20:52:23.081421 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.078738    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 20:52:23.081421 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.078741    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 20:52:23.081421 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.078743    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 20:52:23.081421 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.078746    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 20:52:23.081421 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.078750    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 20:52:23.081913 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.078752    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 20:52:23.081913 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.078755    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 20:52:23.081913 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.078757    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 20:52:23.081913 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.078760    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 20:52:23.081913 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.078762    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 20:52:23.081913 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.078765    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 20:52:23.081913 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.078767    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 20:52:23.081913 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.078769    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 20:52:23.081913 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.078772    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 20:52:23.081913 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.078775    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 20:52:23.081913 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.078777    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 20:52:23.081913 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.078780    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 20:52:23.081913 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.078782    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 20:52:23.081913 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.078785    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 20:52:23.081913 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.078787    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 20:52:23.081913 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.078790    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 20:52:23.081913 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.078792    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 20:52:23.081913 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.078796    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 20:52:23.081913 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.078798    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 20:52:23.082357 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.078801    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 20:52:23.082357 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.078803    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 20:52:23.082357 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.078805    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 20:52:23.082357 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.078808    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 20:52:23.082357 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.078811    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 20:52:23.082357 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.078813    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 20:52:23.082357 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.078815    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 20:52:23.082357 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.078818    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 20:52:23.082357 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.078820    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 20:52:23.082357 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.078827    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 20:52:23.082357 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.078830    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 20:52:23.082357 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.078833    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 20:52:23.082357 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.078836    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 20:52:23.082357 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.078839    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 20:52:23.082357 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.078841    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 20:52:23.082357 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.078844    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 20:52:23.082357 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.078847    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 20:52:23.082357 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.078849    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 20:52:23.082813 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.078854    2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 09 20:52:23.082813 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.078979    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 20:52:23.082813 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.078984    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 20:52:23.082813 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.078987    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 20:52:23.082813 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.078990    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 20:52:23.082813 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.078993    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 20:52:23.082813 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.078996    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 20:52:23.082813 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.078999    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 20:52:23.082813 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.079002    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 20:52:23.082813 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.079005    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 20:52:23.082813 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.079008    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 20:52:23.082813 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.079010    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 20:52:23.082813 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.079013    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 20:52:23.082813 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.079016    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 20:52:23.082813 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.079019    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 20:52:23.083180 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.079021    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 20:52:23.083180 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.079024    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 20:52:23.083180 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.079027    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 20:52:23.083180 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.079029    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 20:52:23.083180 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.079032    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 20:52:23.083180 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.079035    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 20:52:23.083180 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.079038    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 20:52:23.083180 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.079040    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 20:52:23.083180 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.079043    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 20:52:23.083180 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.079046    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 20:52:23.083180 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.079049    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 20:52:23.083180 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.079051    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 20:52:23.083180 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.079054    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 20:52:23.083180 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.079057    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 20:52:23.083180 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.079060    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 20:52:23.083180 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.079062    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 20:52:23.083180 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.079065    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 20:52:23.083180 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.079069    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 20:52:23.083180 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.079072    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 20:52:23.083667 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.079076    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 20:52:23.083667 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.079079    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 20:52:23.083667 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.079082    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 20:52:23.083667 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.079085    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 20:52:23.083667 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.079087    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 20:52:23.083667 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.079090    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 20:52:23.083667 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.079093    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 20:52:23.083667 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.079096    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 20:52:23.083667 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.079098    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 20:52:23.083667 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.079101    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 20:52:23.083667 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.079103    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 20:52:23.083667 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.079106    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 20:52:23.083667 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.079109    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 20:52:23.083667 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.079111    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 20:52:23.083667 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.079114    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 20:52:23.083667 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.079116    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 20:52:23.083667 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.079119    2579 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 20:52:23.083667 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.079121    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 20:52:23.083667 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.079124    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 20:52:23.083667 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.079126    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 20:52:23.084141 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.079129    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 20:52:23.084141 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.079132    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 20:52:23.084141 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.079134    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 20:52:23.084141 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.079137    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 20:52:23.084141 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.079140    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 20:52:23.084141 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.079143    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 20:52:23.084141 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.079146    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 20:52:23.084141 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.079149    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 20:52:23.084141 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.079152    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 20:52:23.084141 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.079154    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 20:52:23.084141 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.079157    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 20:52:23.084141 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.079160    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 20:52:23.084141 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.079162    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 20:52:23.084141 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.079165    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 20:52:23.084141 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.079167    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 20:52:23.084141 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.079170    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 20:52:23.084141 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.079172    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 20:52:23.084141 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.079174    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 20:52:23.084141 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.079177    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 20:52:23.084661 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.079179    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 20:52:23.084661 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.079182    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 20:52:23.084661 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.079185    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 20:52:23.084661 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.079187    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 20:52:23.084661 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.079190    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 20:52:23.084661 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.079192    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 20:52:23.084661 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.079195    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 20:52:23.084661 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.079198    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 20:52:23.084661 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.079201    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 20:52:23.084661 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.079203    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 20:52:23.084661 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.079205    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 20:52:23.084661 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.079208    2579 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 20:52:23.084661 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.079211    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 20:52:23.084661 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.079213    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 20:52:23.084661 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.079217    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 20:52:23.084661 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.079220    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 20:52:23.084661 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.079223    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 20:52:23.084661 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.079226    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 20:52:23.084661 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.079230    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 20:52:23.085097 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.079233    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 20:52:23.085097 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.079235    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 20:52:23.085097 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.079238    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 20:52:23.085097 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.079241    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 20:52:23.085097 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.079243    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 20:52:23.085097 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.079246    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 20:52:23.085097 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.079249    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 20:52:23.085097 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:23.079251    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 20:52:23.085097 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.079256    2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 09 20:52:23.085097 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.079353    2579 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 09 20:52:23.085097 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.084022    2579 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 09 20:52:23.085097 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.085020    2579 server.go:1019] "Starting client certificate rotation"
Sep 09 20:52:23.085402 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.085126    2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 09 20:52:23.085402 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.085161    2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 09 20:52:23.088753 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.088740    2579 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 09 20:52:23.089930 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.089913    2579 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 09 20:52:23.101514 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.101498    2579 log.go:25] "Validated CRI v1 runtime API"
Sep 09 20:52:23.105620 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.105603    2579 log.go:25] "Validated CRI v1 image API"
Sep 09 20:52:23.106574 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.106557    2579 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 09 20:52:23.109125 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.109100    2579 fs.go:135] Filesystem UUIDs: map[34175cf4-0fc7-465f-9d85-83bd4e580352:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 d038e8b2-8b5a-4487-9025-1aacc4802c63:/dev/nvme0n1p4]
Sep 09 20:52:23.109189 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.109125    2579 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 09 20:52:23.115205 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.115098    2579 manager.go:216] Machine: {Timestamp:2026-09-09 20:52:23.113868244 +0000 UTC m=+0.353962353 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3088579 MemoryCapacity:32812150784 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec26663c9b8a98a219801e649ddda0b9 SystemUUID:ec26663c-9b8a-98a2-1980-1e649ddda0b9 BootID:caab0152-1980-40d6-9100-d2feb5a39922 Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406077440 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406073344 Type:vfs Inodes:4005389 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562430976 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:5e:72:85:71:f3 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:5e:72:85:71:f3 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:aa:ac:24:92:c0:e2 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812150784 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 09 20:52:23.115205 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.115203    2579 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 09 20:52:23.115306 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.115275    2579 manager.go:232] Version: {KernelVersion:5.14.0-570.138.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260901-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 09 20:52:23.116482 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.116460    2579 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 09 20:52:23.116632 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.116490    2579 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-138-224.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 09 20:52:23.117352 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.117342    2579 topology_manager.go:138] "Creating topology manager with none policy"
Sep 09 20:52:23.117382 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.117355    2579 container_manager_linux.go:309] "Creating device plugin manager"
Sep 09 20:52:23.117382 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.117362    2579 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 09 20:52:23.118204 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.118191    2579 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 09 20:52:23.120655 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.120395    2579 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 09 20:52:23.126408 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.126395    2579 state_mem.go:36] "Initialized new in-memory state store"
Sep 09 20:52:23.127330 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.127316    2579 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 09 20:52:23.127393 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.127381    2579 kubelet.go:486] "Attempting to sync node with API server"
Sep 09 20:52:23.127455 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.127400    2579 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 09 20:52:23.127455 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.127413    2579 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 09 20:52:23.127455 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.127422    2579 kubelet.go:398] "Adding apiserver pod source"
Sep 09 20:52:23.127455 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.127430    2579 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 09 20:52:23.128036 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.128017    2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 09 20:52:23.128122 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.128045    2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 09 20:52:23.130679 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.130551    2579 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 09 20:52:23.131590 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.131564    2579 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 09 20:52:23.134585 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.134571    2579 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 09 20:52:23.134678 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.134615    2579 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 09 20:52:23.134780 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.134769    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 09 20:52:23.134831 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.134787    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 09 20:52:23.134831 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.134794    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 09 20:52:23.134831 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.134800    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 09 20:52:23.134831 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.134811    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 09 20:52:23.134831 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.134822    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 09 20:52:23.135525 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.135515    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 09 20:52:23.135559 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.135526    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 09 20:52:23.135559 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.135534    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 09 20:52:23.135559 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.135541    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 09 20:52:23.135559 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.135550    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 09 20:52:23.135559 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.135559    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 09 20:52:23.136389 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.136379    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 09 20:52:23.136389 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.136389    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 09 20:52:23.138847 ip-10-0-138-224 kubenswrapper[2579]: E0909 20:52:23.138825    2579 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 09 20:52:23.138971 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.138957    2579 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-224.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 09 20:52:23.139140 ip-10-0-138-224 kubenswrapper[2579]: E0909 20:52:23.139115    2579 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-138-224.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 09 20:52:23.140240 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.140227    2579 server.go:1268] "Started kubelet"
Sep 09 20:52:23.140360 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.140295    2579 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 09 20:52:23.140413 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.140381    2579 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 09 20:52:23.140482 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.140457    2579 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 09 20:52:23.141013 ip-10-0-138-224 systemd[1]: Started Kubernetes Kubelet.
Sep 09 20:52:23.142813 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.142796    2579 server.go:310] "Adding debug handlers to kubelet server"
Sep 09 20:52:23.144663 ip-10-0-138-224 kubenswrapper[2579]: E0909 20:52:23.144638    2579 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 09 20:52:23.144899 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.144887    2579 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 09 20:52:23.146222 ip-10-0-138-224 kubenswrapper[2579]: E0909 20:52:23.145090    2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-224.ec2.internal.18d3c254069cd2be  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-224.ec2.internal,UID:ip-10-0-138-224.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-138-224.ec2.internal,},FirstTimestamp:2026-09-09 20:52:23.14020115 +0000 UTC m=+0.380295260,LastTimestamp:2026-09-09 20:52:23.14020115 +0000 UTC m=+0.380295260,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-224.ec2.internal,}"
Sep 09 20:52:23.149886 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.149868    2579 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 09 20:52:23.149886 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.149884    2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 09 20:52:23.150028 ip-10-0-138-224 kubenswrapper[2579]: E0909 20:52:23.150005    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-224.ec2.internal\" not found"
Sep 09 20:52:23.150879 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.150806    2579 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 09 20:52:23.150879 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.150815    2579 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 09 20:52:23.150879 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.150874    2579 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 09 20:52:23.150879 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.150879    2579 factory.go:55] Registering systemd factory
Sep 09 20:52:23.151118 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.150894    2579 factory.go:223] Registration of the systemd container factory successfully
Sep 09 20:52:23.151118 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.150950    2579 reconstruct.go:97] "Volume reconstruction finished"
Sep 09 20:52:23.151118 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.150957    2579 reconciler.go:29] "Reconciler: start to sync state"
Sep 09 20:52:23.151269 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.151064    2579 factory.go:153] Registering CRI-O factory
Sep 09 20:52:23.151269 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.151206    2579 factory.go:223] Registration of the crio container factory successfully
Sep 09 20:52:23.151361 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.151278    2579 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 09 20:52:23.151361 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.151293    2579 factory.go:103] Registering Raw factory
Sep 09 20:52:23.151361 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.151303    2579 manager.go:1196] Started watching for new ooms in manager
Sep 09 20:52:23.151804 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.151790    2579 manager.go:349] Starting recovery of all containers
Sep 09 20:52:23.154947 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.154924    2579 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-zwsdx"
Sep 09 20:52:23.160640 ip-10-0-138-224 kubenswrapper[2579]: E0909 20:52:23.160601    2579 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 09 20:52:23.160805 ip-10-0-138-224 kubenswrapper[2579]: E0909 20:52:23.160783    2579 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-138-224.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 09 20:52:23.164886 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.164864    2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-zwsdx"
Sep 09 20:52:23.165217 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.165201    2579 manager.go:354] Recovery completed
Sep 09 20:52:23.169340 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.169326    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 20:52:23.171157 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.171138    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-224.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 20:52:23.171222 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.171168    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-224.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 20:52:23.171222 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.171182    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-224.ec2.internal" event="NodeHasSufficientPID"
Sep 09 20:52:23.171706 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.171693    2579 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 09 20:52:23.171706 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.171705    2579 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 09 20:52:23.171824 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.171707    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 20:52:23.171824 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.171721    2579 state_mem.go:36] "Initialized new in-memory state store"
Sep 09 20:52:23.172325 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.172310    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-224.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 20:52:23.172370 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.172337    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-224.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 20:52:23.172370 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.172348    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-224.ec2.internal" event="NodeHasSufficientPID"
Sep 09 20:52:23.173646 ip-10-0-138-224 kubenswrapper[2579]: E0909 20:52:23.173576    2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-224.ec2.internal.18d3c25408752817  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-224.ec2.internal,UID:ip-10-0-138-224.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-138-224.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-138-224.ec2.internal,},FirstTimestamp:2026-09-09 20:52:23.171155991 +0000 UTC m=+0.411250099,LastTimestamp:2026-09-09 20:52:23.171155991 +0000 UTC m=+0.411250099,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-224.ec2.internal,}"
Sep 09 20:52:23.173745 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.173655    2579 policy_none.go:49] "None policy: Start"
Sep 09 20:52:23.173745 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.173670    2579 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 09 20:52:23.173745 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.173682    2579 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 09 20:52:23.175683 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.175672    2579 policy_none.go:47] "Start"
Sep 09 20:52:23.216000 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.215985    2579 manager.go:337] "Starting Device Plugin manager"
Sep 09 20:52:23.231045 ip-10-0-138-224 kubenswrapper[2579]: E0909 20:52:23.216020    2579 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 09 20:52:23.231045 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.216029    2579 server.go:87] "Starting device plugin registration server"
Sep 09 20:52:23.231045 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.216320    2579 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 09 20:52:23.231045 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.216330    2579 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 09 20:52:23.231045 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.216431    2579 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 09 20:52:23.231045 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.216523    2579 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 09 20:52:23.231045 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.216530    2579 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 09 20:52:23.231045 ip-10-0-138-224 kubenswrapper[2579]: E0909 20:52:23.217019    2579 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 09 20:52:23.231045 ip-10-0-138-224 kubenswrapper[2579]: E0909 20:52:23.217062    2579 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-138-224.ec2.internal\" not found"
Sep 09 20:52:23.283478 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.283454    2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 09 20:52:23.284773 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.284759    2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 09 20:52:23.284828 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.284808    2579 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 09 20:52:23.284875 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.284832    2579 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 09 20:52:23.284875 ip-10-0-138-224 kubenswrapper[2579]: E0909 20:52:23.284870    2579 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 09 20:52:23.287270 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.287254    2579 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 20:52:23.317307 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.317243    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 20:52:23.319942 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.319925    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-224.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 20:52:23.320028 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.319961    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-224.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 20:52:23.320028 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.319978    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-224.ec2.internal" event="NodeHasSufficientPID"
Sep 09 20:52:23.320028 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.320007    2579 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-138-224.ec2.internal"
Sep 09 20:52:23.329002 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.328983    2579 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-138-224.ec2.internal"
Sep 09 20:52:23.329075 ip-10-0-138-224 kubenswrapper[2579]: E0909 20:52:23.329007    2579 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-138-224.ec2.internal\": node \"ip-10-0-138-224.ec2.internal\" not found"
Sep 09 20:52:23.348965 ip-10-0-138-224 kubenswrapper[2579]: E0909 20:52:23.348949    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-224.ec2.internal\" not found"
Sep 09 20:52:23.385885 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.385862    2579 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-224.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-138-224.ec2.internal"]
Sep 09 20:52:23.385973 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.385937    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 20:52:23.387356 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.387335    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-224.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 20:52:23.387415 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.387363    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-224.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 20:52:23.387415 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.387373    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-224.ec2.internal" event="NodeHasSufficientPID"
Sep 09 20:52:23.389609 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.389596    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 20:52:23.389739 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.389726    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-224.ec2.internal"
Sep 09 20:52:23.389792 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.389766    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 20:52:23.390323 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.390308    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-224.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 20:52:23.390414 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.390334    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-224.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 20:52:23.390414 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.390350    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-224.ec2.internal" event="NodeHasSufficientPID"
Sep 09 20:52:23.390743 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.390726    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-224.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 20:52:23.390837 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.390753    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-224.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 20:52:23.390837 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.390780    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-224.ec2.internal" event="NodeHasSufficientPID"
Sep 09 20:52:23.392672 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.392658    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-224.ec2.internal"
Sep 09 20:52:23.392742 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.392693    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 20:52:23.393383 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.393367    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-224.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 20:52:23.393498 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.393395    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-224.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 20:52:23.393498 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.393410    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-224.ec2.internal" event="NodeHasSufficientPID"
Sep 09 20:52:23.422988 ip-10-0-138-224 kubenswrapper[2579]: E0909 20:52:23.422970    2579 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-224.ec2.internal\" not found" node="ip-10-0-138-224.ec2.internal"
Sep 09 20:52:23.427170 ip-10-0-138-224 kubenswrapper[2579]: E0909 20:52:23.427155    2579 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-224.ec2.internal\" not found" node="ip-10-0-138-224.ec2.internal"
Sep 09 20:52:23.449156 ip-10-0-138-224 kubenswrapper[2579]: E0909 20:52:23.449138    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-224.ec2.internal\" not found"
Sep 09 20:52:23.452419 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.452402    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a125a7c81ad75870351cc3dc16d80838-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-224.ec2.internal\" (UID: \"a125a7c81ad75870351cc3dc16d80838\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-224.ec2.internal"
Sep 09 20:52:23.452487 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.452428    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7e4d14d2e217a05b09faab3ef55f4d0f-config\") pod \"kube-apiserver-proxy-ip-10-0-138-224.ec2.internal\" (UID: \"7e4d14d2e217a05b09faab3ef55f4d0f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-224.ec2.internal"
Sep 09 20:52:23.452487 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.452463    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a125a7c81ad75870351cc3dc16d80838-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-224.ec2.internal\" (UID: \"a125a7c81ad75870351cc3dc16d80838\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-224.ec2.internal"
Sep 09 20:52:23.549397 ip-10-0-138-224 kubenswrapper[2579]: E0909 20:52:23.549374    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-224.ec2.internal\" not found"
Sep 09 20:52:23.552666 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.552650    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a125a7c81ad75870351cc3dc16d80838-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-224.ec2.internal\" (UID: \"a125a7c81ad75870351cc3dc16d80838\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-224.ec2.internal"
Sep 09 20:52:23.552735 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.552674    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a125a7c81ad75870351cc3dc16d80838-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-224.ec2.internal\" (UID: \"a125a7c81ad75870351cc3dc16d80838\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-224.ec2.internal"
Sep 09 20:52:23.552735 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.552691    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7e4d14d2e217a05b09faab3ef55f4d0f-config\") pod \"kube-apiserver-proxy-ip-10-0-138-224.ec2.internal\" (UID: \"7e4d14d2e217a05b09faab3ef55f4d0f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-224.ec2.internal"
Sep 09 20:52:23.552735 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.552719    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7e4d14d2e217a05b09faab3ef55f4d0f-config\") pod \"kube-apiserver-proxy-ip-10-0-138-224.ec2.internal\" (UID: \"7e4d14d2e217a05b09faab3ef55f4d0f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-224.ec2.internal"
Sep 09 20:52:23.552871 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.552759    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a125a7c81ad75870351cc3dc16d80838-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-224.ec2.internal\" (UID: \"a125a7c81ad75870351cc3dc16d80838\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-224.ec2.internal"
Sep 09 20:52:23.552871 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.552756    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a125a7c81ad75870351cc3dc16d80838-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-224.ec2.internal\" (UID: \"a125a7c81ad75870351cc3dc16d80838\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-224.ec2.internal"
Sep 09 20:52:23.649610 ip-10-0-138-224 kubenswrapper[2579]: E0909 20:52:23.649554    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-224.ec2.internal\" not found"
Sep 09 20:52:23.725988 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.725968    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-224.ec2.internal"
Sep 09 20:52:23.729595 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.729579    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-224.ec2.internal"
Sep 09 20:52:23.750699 ip-10-0-138-224 kubenswrapper[2579]: E0909 20:52:23.750527    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-224.ec2.internal\" not found"
Sep 09 20:52:23.851008 ip-10-0-138-224 kubenswrapper[2579]: E0909 20:52:23.850968    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-224.ec2.internal\" not found"
Sep 09 20:52:23.949259 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.949212    2579 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 20:52:23.950792 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.950779    2579 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-224.ec2.internal"
Sep 09 20:52:23.962898 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.962880    2579 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 09 20:52:23.964300 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.964289    2579 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-224.ec2.internal"
Sep 09 20:52:23.971566 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:23.971554    2579 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 09 20:52:24.084984 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.084954    2579 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 09 20:52:24.085434 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.085118    2579 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 09 20:52:24.085434 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.085128    2579 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 09 20:52:24.128267 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.128244    2579 apiserver.go:52] "Watching apiserver"
Sep 09 20:52:24.144783 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.144453    2579 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 09 20:52:24.145419 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.145395    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-88pfs","openshift-multus/multus-additional-cni-plugins-9rprp","openshift-multus/multus-jbs9t","openshift-network-diagnostics/network-check-target-b6qr7","openshift-ovn-kubernetes/ovnkube-node-9dcqf","kube-system/global-pull-secret-syncer-gz9zx","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pdpk7","openshift-cluster-node-tuning-operator/tuned-s98q4","openshift-image-registry/node-ca-tml69","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-224.ec2.internal","openshift-multus/network-metrics-daemon-ws8hc","openshift-network-operator/iptables-alerter-kfpbw","kube-system/konnectivity-agent-kwvw7","kube-system/kube-apiserver-proxy-ip-10-0-138-224.ec2.internal"]
Sep 09 20:52:24.148037 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.148011    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-88pfs"
Sep 09 20:52:24.150112 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.150096    2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 09 20:52:24.150242 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.150226    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-9rprp"
Sep 09 20:52:24.150748 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.150733    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 09 20:52:24.150748 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.150740    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 09 20:52:24.150884 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.150796    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-hvcgc\""
Sep 09 20:52:24.152317 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.152301    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-jbs9t"
Sep 09 20:52:24.152589 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.152568    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 09 20:52:24.152589 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.152581    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 09 20:52:24.152775 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.152681    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-d6fcb\""
Sep 09 20:52:24.152842 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.152787    2579 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 20:52:24.153074 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.153057    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 09 20:52:24.153325 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.153305    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 09 20:52:24.153452 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.153338    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 09 20:52:24.154524 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.154509    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b6qr7"
Sep 09 20:52:24.154603 ip-10-0-138-224 kubenswrapper[2579]: E0909 20:52:24.154587    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b6qr7" podUID="e1193e63-dea9-41f4-a6ab-748d8ae109e8"
Sep 09 20:52:24.154857 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.154837    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 09 20:52:24.154857 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.154850    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ddb26243-140d-4ac9-8a3d-4079559bd0e7-tmp-dir\") pod \"node-resolver-88pfs\" (UID: \"ddb26243-140d-4ac9-8a3d-4079559bd0e7\") " pod="openshift-dns/node-resolver-88pfs"
Sep 09 20:52:24.154984 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.154879    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fq6gl\" (UniqueName: \"kubernetes.io/projected/ddb26243-140d-4ac9-8a3d-4079559bd0e7-kube-api-access-fq6gl\") pod \"node-resolver-88pfs\" (UID: \"ddb26243-140d-4ac9-8a3d-4079559bd0e7\") " pod="openshift-dns/node-resolver-88pfs"
Sep 09 20:52:24.154984 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.154904    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/dfe1a979-9dff-419b-b517-93e03c19c040-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9rprp\" (UID: \"dfe1a979-9dff-419b-b517-93e03c19c040\") " pod="openshift-multus/multus-additional-cni-plugins-9rprp"
Sep 09 20:52:24.154984 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.154941    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/22f66b47-36ad-4226-b0f9-ad4075eaaa2d-multus-cni-dir\") pod \"multus-jbs9t\" (UID: \"22f66b47-36ad-4226-b0f9-ad4075eaaa2d\") " pod="openshift-multus/multus-jbs9t"
Sep 09 20:52:24.155114 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.154984    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/22f66b47-36ad-4226-b0f9-ad4075eaaa2d-host-run-netns\") pod \"multus-jbs9t\" (UID: \"22f66b47-36ad-4226-b0f9-ad4075eaaa2d\") " pod="openshift-multus/multus-jbs9t"
Sep 09 20:52:24.155114 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.155005    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dfe1a979-9dff-419b-b517-93e03c19c040-system-cni-dir\") pod \"multus-additional-cni-plugins-9rprp\" (UID: \"dfe1a979-9dff-419b-b517-93e03c19c040\") " pod="openshift-multus/multus-additional-cni-plugins-9rprp"
Sep 09 20:52:24.155114 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.155025    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/22f66b47-36ad-4226-b0f9-ad4075eaaa2d-host-var-lib-cni-multus\") pod \"multus-jbs9t\" (UID: \"22f66b47-36ad-4226-b0f9-ad4075eaaa2d\") " pod="openshift-multus/multus-jbs9t"
Sep 09 20:52:24.155114 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.155039    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-qlc2g\""
Sep 09 20:52:24.155114 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.155049    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/22f66b47-36ad-4226-b0f9-ad4075eaaa2d-multus-daemon-config\") pod \"multus-jbs9t\" (UID: \"22f66b47-36ad-4226-b0f9-ad4075eaaa2d\") " pod="openshift-multus/multus-jbs9t"
Sep 09 20:52:24.155114 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.155072    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/22f66b47-36ad-4226-b0f9-ad4075eaaa2d-host-run-multus-certs\") pod \"multus-jbs9t\" (UID: \"22f66b47-36ad-4226-b0f9-ad4075eaaa2d\") " pod="openshift-multus/multus-jbs9t"
Sep 09 20:52:24.155360 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.155118    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/dfe1a979-9dff-419b-b517-93e03c19c040-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9rprp\" (UID: \"dfe1a979-9dff-419b-b517-93e03c19c040\") " pod="openshift-multus/multus-additional-cni-plugins-9rprp"
Sep 09 20:52:24.155360 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.155177    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/22f66b47-36ad-4226-b0f9-ad4075eaaa2d-cni-binary-copy\") pod \"multus-jbs9t\" (UID: \"22f66b47-36ad-4226-b0f9-ad4075eaaa2d\") " pod="openshift-multus/multus-jbs9t"
Sep 09 20:52:24.155360 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.155196    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/22f66b47-36ad-4226-b0f9-ad4075eaaa2d-host-var-lib-cni-bin\") pod \"multus-jbs9t\" (UID: \"22f66b47-36ad-4226-b0f9-ad4075eaaa2d\") " pod="openshift-multus/multus-jbs9t"
Sep 09 20:52:24.155360 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.155215    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/22f66b47-36ad-4226-b0f9-ad4075eaaa2d-multus-conf-dir\") pod \"multus-jbs9t\" (UID: \"22f66b47-36ad-4226-b0f9-ad4075eaaa2d\") " pod="openshift-multus/multus-jbs9t"
Sep 09 20:52:24.155360 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.155238    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dfe1a979-9dff-419b-b517-93e03c19c040-cnibin\") pod \"multus-additional-cni-plugins-9rprp\" (UID: \"dfe1a979-9dff-419b-b517-93e03c19c040\") " pod="openshift-multus/multus-additional-cni-plugins-9rprp"
Sep 09 20:52:24.155360 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.155257    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/dfe1a979-9dff-419b-b517-93e03c19c040-cni-binary-copy\") pod \"multus-additional-cni-plugins-9rprp\" (UID: \"dfe1a979-9dff-419b-b517-93e03c19c040\") " pod="openshift-multus/multus-additional-cni-plugins-9rprp"
Sep 09 20:52:24.155360 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.155271    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/22f66b47-36ad-4226-b0f9-ad4075eaaa2d-hostroot\") pod \"multus-jbs9t\" (UID: \"22f66b47-36ad-4226-b0f9-ad4075eaaa2d\") " pod="openshift-multus/multus-jbs9t"
Sep 09 20:52:24.155360 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.155319    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/22f66b47-36ad-4226-b0f9-ad4075eaaa2d-etc-kubernetes\") pod \"multus-jbs9t\" (UID: \"22f66b47-36ad-4226-b0f9-ad4075eaaa2d\") " pod="openshift-multus/multus-jbs9t"
Sep 09 20:52:24.155740 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.155373    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/22f66b47-36ad-4226-b0f9-ad4075eaaa2d-system-cni-dir\") pod \"multus-jbs9t\" (UID: \"22f66b47-36ad-4226-b0f9-ad4075eaaa2d\") " pod="openshift-multus/multus-jbs9t"
Sep 09 20:52:24.155740 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.155400    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/22f66b47-36ad-4226-b0f9-ad4075eaaa2d-cnibin\") pod \"multus-jbs9t\" (UID: \"22f66b47-36ad-4226-b0f9-ad4075eaaa2d\") " pod="openshift-multus/multus-jbs9t"
Sep 09 20:52:24.155740 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.155419    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prbqs\" (UniqueName: \"kubernetes.io/projected/22f66b47-36ad-4226-b0f9-ad4075eaaa2d-kube-api-access-prbqs\") pod \"multus-jbs9t\" (UID: \"22f66b47-36ad-4226-b0f9-ad4075eaaa2d\") " pod="openshift-multus/multus-jbs9t"
Sep 09 20:52:24.155740 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.155460    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dfe1a979-9dff-419b-b517-93e03c19c040-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9rprp\" (UID: \"dfe1a979-9dff-419b-b517-93e03c19c040\") " pod="openshift-multus/multus-additional-cni-plugins-9rprp"
Sep 09 20:52:24.155740 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.155479    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ggfx\" (UniqueName: \"kubernetes.io/projected/dfe1a979-9dff-419b-b517-93e03c19c040-kube-api-access-9ggfx\") pod \"multus-additional-cni-plugins-9rprp\" (UID: \"dfe1a979-9dff-419b-b517-93e03c19c040\") " pod="openshift-multus/multus-additional-cni-plugins-9rprp"
Sep 09 20:52:24.155740 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.155500    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/22f66b47-36ad-4226-b0f9-ad4075eaaa2d-host-run-k8s-cni-cncf-io\") pod \"multus-jbs9t\" (UID: \"22f66b47-36ad-4226-b0f9-ad4075eaaa2d\") " pod="openshift-multus/multus-jbs9t"
Sep 09 20:52:24.155740 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.155514    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ddb26243-140d-4ac9-8a3d-4079559bd0e7-hosts-file\") pod \"node-resolver-88pfs\" (UID: \"ddb26243-140d-4ac9-8a3d-4079559bd0e7\") " pod="openshift-dns/node-resolver-88pfs"
Sep 09 20:52:24.155740 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.155530    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/22f66b47-36ad-4226-b0f9-ad4075eaaa2d-multus-socket-dir-parent\") pod \"multus-jbs9t\" (UID: \"22f66b47-36ad-4226-b0f9-ad4075eaaa2d\") " pod="openshift-multus/multus-jbs9t"
Sep 09 20:52:24.155740 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.155545    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/22f66b47-36ad-4226-b0f9-ad4075eaaa2d-host-var-lib-kubelet\") pod \"multus-jbs9t\" (UID: \"22f66b47-36ad-4226-b0f9-ad4075eaaa2d\") " pod="openshift-multus/multus-jbs9t"
Sep 09 20:52:24.156733 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.156719    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-9dcqf"
Sep 09 20:52:24.158905 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.158888    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gz9zx"
Sep 09 20:52:24.159001 ip-10-0-138-224 kubenswrapper[2579]: E0909 20:52:24.158974    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gz9zx" podUID="732d9d71-d680-4291-aaee-028307ef79dc"
Sep 09 20:52:24.159343 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.159322    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 09 20:52:24.159405 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.159324    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-m4s59\""
Sep 09 20:52:24.159405 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.159389    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 09 20:52:24.159522 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.159472    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 09 20:52:24.159522 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.159493    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 09 20:52:24.159628 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.159532    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 09 20:52:24.159707 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.159688    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 09 20:52:24.161251 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.161236    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pdpk7"
Sep 09 20:52:24.163078 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.163063    2579 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 09 20:52:24.163519 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.163505    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-s98q4"
Sep 09 20:52:24.164212 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.164194    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 09 20:52:24.164293 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.164248    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-kf52q\""
Sep 09 20:52:24.164293 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.164278    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 09 20:52:24.164417 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.164403    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 09 20:52:24.165893 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.165876    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-tml69"
Sep 09 20:52:24.166786 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.166768    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 09 20:52:24.166935 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.166920    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-8n2qj\""
Sep 09 20:52:24.166987 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.166922    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 09 20:52:24.167501 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.167478    2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-08 20:47:23 +0000 UTC" deadline="2028-04-23 23:08:05.777930021 +0000 UTC"
Sep 09 20:52:24.167554 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.167503    2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14210h15m41.610430813s"
Sep 09 20:52:24.168358 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.168167    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ws8hc"
Sep 09 20:52:24.168358 ip-10-0-138-224 kubenswrapper[2579]: E0909 20:52:24.168230    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ws8hc" podUID="c6fd107c-ea2f-4e85-9a32-1e11f0858fd3"
Sep 09 20:52:24.168358 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.168247    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 09 20:52:24.168599 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.168582    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-6k6gn\""
Sep 09 20:52:24.168647 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.168609    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 09 20:52:24.168862 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.168845    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 09 20:52:24.172112 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.172095    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-kfpbw"
Sep 09 20:52:24.174350 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.174334    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-kwvw7"
Sep 09 20:52:24.174655 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.174639    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-h24zj\""
Sep 09 20:52:24.174655 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.174648    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 09 20:52:24.174783 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.174678    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 09 20:52:24.174783 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.174743    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 09 20:52:24.176735 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.176719    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 09 20:52:24.176735 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.176731    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-mmtbs\""
Sep 09 20:52:24.177065 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.177047    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 09 20:52:24.189579 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.189563    2579 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-s246k"
Sep 09 20:52:24.195415 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.195387    2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-s246k"
Sep 09 20:52:24.252634 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.252617    2579 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 09 20:52:24.255855 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.255835    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b2481c54-52e5-44b3-a24a-09b4dd9f575b-ovnkube-config\") pod \"ovnkube-node-9dcqf\" (UID: \"b2481c54-52e5-44b3-a24a-09b4dd9f575b\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dcqf"
Sep 09 20:52:24.255949 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.255861    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/10f78cd8-5259-4024-ae5f-68bfa6a0626f-sys\") pod \"tuned-s98q4\" (UID: \"10f78cd8-5259-4024-ae5f-68bfa6a0626f\") " pod="openshift-cluster-node-tuning-operator/tuned-s98q4"
Sep 09 20:52:24.255949 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.255897    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/28da7add-257c-49f6-a846-6e8116b68d4c-host-slash\") pod \"iptables-alerter-kfpbw\" (UID: \"28da7add-257c-49f6-a846-6e8116b68d4c\") " pod="openshift-network-operator/iptables-alerter-kfpbw"
Sep 09 20:52:24.255949 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.255931    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/dfe1a979-9dff-419b-b517-93e03c19c040-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9rprp\" (UID: \"dfe1a979-9dff-419b-b517-93e03c19c040\") " pod="openshift-multus/multus-additional-cni-plugins-9rprp"
Sep 09 20:52:24.255949 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.255950    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/22f66b47-36ad-4226-b0f9-ad4075eaaa2d-host-var-lib-cni-bin\") pod \"multus-jbs9t\" (UID: \"22f66b47-36ad-4226-b0f9-ad4075eaaa2d\") " pod="openshift-multus/multus-jbs9t"
Sep 09 20:52:24.256108 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.255968    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/22f66b47-36ad-4226-b0f9-ad4075eaaa2d-multus-conf-dir\") pod \"multus-jbs9t\" (UID: \"22f66b47-36ad-4226-b0f9-ad4075eaaa2d\") " pod="openshift-multus/multus-jbs9t"
Sep 09 20:52:24.256108 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.255995    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a55d0249-76a1-42d0-8e06-8c4fa1e557f8-registration-dir\") pod \"aws-ebs-csi-driver-node-pdpk7\" (UID: \"a55d0249-76a1-42d0-8e06-8c4fa1e557f8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pdpk7"
Sep 09 20:52:24.256108 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.256017    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tnkxj\" (UniqueName: \"kubernetes.io/projected/28da7add-257c-49f6-a846-6e8116b68d4c-kube-api-access-tnkxj\") pod \"iptables-alerter-kfpbw\" (UID: \"28da7add-257c-49f6-a846-6e8116b68d4c\") " pod="openshift-network-operator/iptables-alerter-kfpbw"
Sep 09 20:52:24.256108 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.256020    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/22f66b47-36ad-4226-b0f9-ad4075eaaa2d-host-var-lib-cni-bin\") pod \"multus-jbs9t\" (UID: \"22f66b47-36ad-4226-b0f9-ad4075eaaa2d\") " pod="openshift-multus/multus-jbs9t"
Sep 09 20:52:24.256108 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.256040    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dfe1a979-9dff-419b-b517-93e03c19c040-cnibin\") pod \"multus-additional-cni-plugins-9rprp\" (UID: \"dfe1a979-9dff-419b-b517-93e03c19c040\") " pod="openshift-multus/multus-additional-cni-plugins-9rprp"
Sep 09 20:52:24.256108 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.256077    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/dfe1a979-9dff-419b-b517-93e03c19c040-cni-binary-copy\") pod \"multus-additional-cni-plugins-9rprp\" (UID: \"dfe1a979-9dff-419b-b517-93e03c19c040\") " pod="openshift-multus/multus-additional-cni-plugins-9rprp"
Sep 09 20:52:24.256108 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.256101    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9075ffbd-9633-49ac-b6aa-f410776c4ad6-konnectivity-ca\") pod \"konnectivity-agent-kwvw7\" (UID: \"9075ffbd-9633-49ac-b6aa-f410776c4ad6\") " pod="kube-system/konnectivity-agent-kwvw7"
Sep 09 20:52:24.256108 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.256105    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/22f66b47-36ad-4226-b0f9-ad4075eaaa2d-multus-conf-dir\") pod \"multus-jbs9t\" (UID: \"22f66b47-36ad-4226-b0f9-ad4075eaaa2d\") " pod="openshift-multus/multus-jbs9t"
Sep 09 20:52:24.256457 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.256123    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c6fd107c-ea2f-4e85-9a32-1e11f0858fd3-metrics-certs\") pod \"network-metrics-daemon-ws8hc\" (UID: \"c6fd107c-ea2f-4e85-9a32-1e11f0858fd3\") " pod="openshift-multus/network-metrics-daemon-ws8hc"
Sep 09 20:52:24.256457 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.256147    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dfe1a979-9dff-419b-b517-93e03c19c040-cnibin\") pod \"multus-additional-cni-plugins-9rprp\" (UID: \"dfe1a979-9dff-419b-b517-93e03c19c040\") " pod="openshift-multus/multus-additional-cni-plugins-9rprp"
Sep 09 20:52:24.256457 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.256161    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/22f66b47-36ad-4226-b0f9-ad4075eaaa2d-cnibin\") pod \"multus-jbs9t\" (UID: \"22f66b47-36ad-4226-b0f9-ad4075eaaa2d\") " pod="openshift-multus/multus-jbs9t"
Sep 09 20:52:24.256457 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.256179    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prbqs\" (UniqueName: \"kubernetes.io/projected/22f66b47-36ad-4226-b0f9-ad4075eaaa2d-kube-api-access-prbqs\") pod \"multus-jbs9t\" (UID: \"22f66b47-36ad-4226-b0f9-ad4075eaaa2d\") " pod="openshift-multus/multus-jbs9t"
Sep 09 20:52:24.256457 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.256313    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b2481c54-52e5-44b3-a24a-09b4dd9f575b-run-openvswitch\") pod \"ovnkube-node-9dcqf\" (UID: \"b2481c54-52e5-44b3-a24a-09b4dd9f575b\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dcqf"
Sep 09 20:52:24.256457 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.256323    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/22f66b47-36ad-4226-b0f9-ad4075eaaa2d-cnibin\") pod \"multus-jbs9t\" (UID: \"22f66b47-36ad-4226-b0f9-ad4075eaaa2d\") " pod="openshift-multus/multus-jbs9t"
Sep 09 20:52:24.256457 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.256348    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a55d0249-76a1-42d0-8e06-8c4fa1e557f8-kubelet-dir\") pod \"aws-ebs-csi-driver-node-pdpk7\" (UID: \"a55d0249-76a1-42d0-8e06-8c4fa1e557f8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pdpk7"
Sep 09 20:52:24.256457 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.256375    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/10f78cd8-5259-4024-ae5f-68bfa6a0626f-etc-tuned\") pod \"tuned-s98q4\" (UID: \"10f78cd8-5259-4024-ae5f-68bfa6a0626f\") " pod="openshift-cluster-node-tuning-operator/tuned-s98q4"
Sep 09 20:52:24.256457 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.256401    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/22f66b47-36ad-4226-b0f9-ad4075eaaa2d-host-run-k8s-cni-cncf-io\") pod \"multus-jbs9t\" (UID: \"22f66b47-36ad-4226-b0f9-ad4075eaaa2d\") " pod="openshift-multus/multus-jbs9t"
Sep 09 20:52:24.256457 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.256427    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/10f78cd8-5259-4024-ae5f-68bfa6a0626f-etc-sysctl-d\") pod \"tuned-s98q4\" (UID: \"10f78cd8-5259-4024-ae5f-68bfa6a0626f\") " pod="openshift-cluster-node-tuning-operator/tuned-s98q4"
Sep 09 20:52:24.256898 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.256488    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/22f66b47-36ad-4226-b0f9-ad4075eaaa2d-multus-socket-dir-parent\") pod \"multus-jbs9t\" (UID: \"22f66b47-36ad-4226-b0f9-ad4075eaaa2d\") " pod="openshift-multus/multus-jbs9t"
Sep 09 20:52:24.256898 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.256533    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/22f66b47-36ad-4226-b0f9-ad4075eaaa2d-host-run-k8s-cni-cncf-io\") pod \"multus-jbs9t\" (UID: \"22f66b47-36ad-4226-b0f9-ad4075eaaa2d\") " pod="openshift-multus/multus-jbs9t"
Sep 09 20:52:24.256898 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.256543    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fq6gl\" (UniqueName: \"kubernetes.io/projected/ddb26243-140d-4ac9-8a3d-4079559bd0e7-kube-api-access-fq6gl\") pod \"node-resolver-88pfs\" (UID: \"ddb26243-140d-4ac9-8a3d-4079559bd0e7\") " pod="openshift-dns/node-resolver-88pfs"
Sep 09 20:52:24.256898 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.256583    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/22f66b47-36ad-4226-b0f9-ad4075eaaa2d-multus-socket-dir-parent\") pod \"multus-jbs9t\" (UID: \"22f66b47-36ad-4226-b0f9-ad4075eaaa2d\") " pod="openshift-multus/multus-jbs9t"
Sep 09 20:52:24.256898 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.256581    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b2481c54-52e5-44b3-a24a-09b4dd9f575b-host-kubelet\") pod \"ovnkube-node-9dcqf\" (UID: \"b2481c54-52e5-44b3-a24a-09b4dd9f575b\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dcqf"
Sep 09 20:52:24.256898 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.256696    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b2481c54-52e5-44b3-a24a-09b4dd9f575b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-9dcqf\" (UID: \"b2481c54-52e5-44b3-a24a-09b4dd9f575b\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dcqf"
Sep 09 20:52:24.256898 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.256747    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b2481c54-52e5-44b3-a24a-09b4dd9f575b-ovnkube-script-lib\") pod \"ovnkube-node-9dcqf\" (UID: \"b2481c54-52e5-44b3-a24a-09b4dd9f575b\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dcqf"
Sep 09 20:52:24.256898 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.256781    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a55d0249-76a1-42d0-8e06-8c4fa1e557f8-etc-selinux\") pod \"aws-ebs-csi-driver-node-pdpk7\" (UID: \"a55d0249-76a1-42d0-8e06-8c4fa1e557f8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pdpk7"
Sep 09 20:52:24.256898 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.256806    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/10f78cd8-5259-4024-ae5f-68bfa6a0626f-etc-systemd\") pod \"tuned-s98q4\" (UID: \"10f78cd8-5259-4024-ae5f-68bfa6a0626f\") " pod="openshift-cluster-node-tuning-operator/tuned-s98q4"
Sep 09 20:52:24.256898 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.256826    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/10f78cd8-5259-4024-ae5f-68bfa6a0626f-lib-modules\") pod \"tuned-s98q4\" (UID: \"10f78cd8-5259-4024-ae5f-68bfa6a0626f\") " pod="openshift-cluster-node-tuning-operator/tuned-s98q4"
Sep 09 20:52:24.256898 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.256849    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b2481c54-52e5-44b3-a24a-09b4dd9f575b-host-run-ovn-kubernetes\") pod \"ovnkube-node-9dcqf\" (UID: \"b2481c54-52e5-44b3-a24a-09b4dd9f575b\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dcqf"
Sep 09 20:52:24.256898 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.256876    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/22f66b47-36ad-4226-b0f9-ad4075eaaa2d-host-run-netns\") pod \"multus-jbs9t\" (UID: \"22f66b47-36ad-4226-b0f9-ad4075eaaa2d\") " pod="openshift-multus/multus-jbs9t"
Sep 09 20:52:24.256898 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.256916    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/732d9d71-d680-4291-aaee-028307ef79dc-kubelet-config\") pod \"global-pull-secret-syncer-gz9zx\" (UID: \"732d9d71-d680-4291-aaee-028307ef79dc\") " pod="kube-system/global-pull-secret-syncer-gz9zx"
Sep 09 20:52:24.257500 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.256971    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/22f66b47-36ad-4226-b0f9-ad4075eaaa2d-host-run-netns\") pod \"multus-jbs9t\" (UID: \"22f66b47-36ad-4226-b0f9-ad4075eaaa2d\") " pod="openshift-multus/multus-jbs9t"
Sep 09 20:52:24.257500 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.257015    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/169cade8-7395-4b62-9544-ff0c47022a89-host\") pod \"node-ca-tml69\" (UID: \"169cade8-7395-4b62-9544-ff0c47022a89\") " pod="openshift-image-registry/node-ca-tml69"
Sep 09 20:52:24.257500 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.257039    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b2481c54-52e5-44b3-a24a-09b4dd9f575b-systemd-units\") pod \"ovnkube-node-9dcqf\" (UID: \"b2481c54-52e5-44b3-a24a-09b4dd9f575b\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dcqf"
Sep 09 20:52:24.257500 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.257058    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/10f78cd8-5259-4024-ae5f-68bfa6a0626f-etc-sysctl-conf\") pod \"tuned-s98q4\" (UID: \"10f78cd8-5259-4024-ae5f-68bfa6a0626f\") " pod="openshift-cluster-node-tuning-operator/tuned-s98q4"
Sep 09 20:52:24.257500 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.257072    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/10f78cd8-5259-4024-ae5f-68bfa6a0626f-var-lib-kubelet\") pod \"tuned-s98q4\" (UID: \"10f78cd8-5259-4024-ae5f-68bfa6a0626f\") " pod="openshift-cluster-node-tuning-operator/tuned-s98q4"
Sep 09 20:52:24.257500 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.257101    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/10f78cd8-5259-4024-ae5f-68bfa6a0626f-host\") pod \"tuned-s98q4\" (UID: \"10f78cd8-5259-4024-ae5f-68bfa6a0626f\") " pod="openshift-cluster-node-tuning-operator/tuned-s98q4"
Sep 09 20:52:24.257500 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.257133    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9075ffbd-9633-49ac-b6aa-f410776c4ad6-agent-certs\") pod \"konnectivity-agent-kwvw7\" (UID: \"9075ffbd-9633-49ac-b6aa-f410776c4ad6\") " pod="kube-system/konnectivity-agent-kwvw7"
Sep 09 20:52:24.257500 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.257153    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b2481c54-52e5-44b3-a24a-09b4dd9f575b-run-systemd\") pod \"ovnkube-node-9dcqf\" (UID: \"b2481c54-52e5-44b3-a24a-09b4dd9f575b\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dcqf"
Sep 09 20:52:24.257500 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.257178    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b2481c54-52e5-44b3-a24a-09b4dd9f575b-etc-openvswitch\") pod \"ovnkube-node-9dcqf\" (UID: \"b2481c54-52e5-44b3-a24a-09b4dd9f575b\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dcqf"
Sep 09 20:52:24.257500 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.257195    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/10f78cd8-5259-4024-ae5f-68bfa6a0626f-etc-sysconfig\") pod \"tuned-s98q4\" (UID: \"10f78cd8-5259-4024-ae5f-68bfa6a0626f\") " pod="openshift-cluster-node-tuning-operator/tuned-s98q4"
Sep 09 20:52:24.257500 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.257209    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/10f78cd8-5259-4024-ae5f-68bfa6a0626f-etc-kubernetes\") pod \"tuned-s98q4\" (UID: \"10f78cd8-5259-4024-ae5f-68bfa6a0626f\") " pod="openshift-cluster-node-tuning-operator/tuned-s98q4"
Sep 09 20:52:24.257500 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.257261    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmgq4\" (UniqueName: \"kubernetes.io/projected/10f78cd8-5259-4024-ae5f-68bfa6a0626f-kube-api-access-gmgq4\") pod \"tuned-s98q4\" (UID: \"10f78cd8-5259-4024-ae5f-68bfa6a0626f\") " pod="openshift-cluster-node-tuning-operator/tuned-s98q4"
Sep 09 20:52:24.257500 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.257279    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/22f66b47-36ad-4226-b0f9-ad4075eaaa2d-host-run-multus-certs\") pod \"multus-jbs9t\" (UID: \"22f66b47-36ad-4226-b0f9-ad4075eaaa2d\") " pod="openshift-multus/multus-jbs9t"
Sep 09 20:52:24.257500 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.257294    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwfng\" (UniqueName: \"kubernetes.io/projected/a55d0249-76a1-42d0-8e06-8c4fa1e557f8-kube-api-access-pwfng\") pod \"aws-ebs-csi-driver-node-pdpk7\" (UID: \"a55d0249-76a1-42d0-8e06-8c4fa1e557f8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pdpk7"
Sep 09 20:52:24.257500 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.257310    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/10f78cd8-5259-4024-ae5f-68bfa6a0626f-run\") pod \"tuned-s98q4\" (UID: \"10f78cd8-5259-4024-ae5f-68bfa6a0626f\") " pod="openshift-cluster-node-tuning-operator/tuned-s98q4"
Sep 09 20:52:24.257500 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.257333    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/22f66b47-36ad-4226-b0f9-ad4075eaaa2d-cni-binary-copy\") pod \"multus-jbs9t\" (UID: \"22f66b47-36ad-4226-b0f9-ad4075eaaa2d\") " pod="openshift-multus/multus-jbs9t"
Sep 09 20:52:24.257500 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.257335    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/22f66b47-36ad-4226-b0f9-ad4075eaaa2d-host-run-multus-certs\") pod \"multus-jbs9t\" (UID: \"22f66b47-36ad-4226-b0f9-ad4075eaaa2d\") " pod="openshift-multus/multus-jbs9t"
Sep 09 20:52:24.258322 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.257355    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/732d9d71-d680-4291-aaee-028307ef79dc-original-pull-secret\") pod \"global-pull-secret-syncer-gz9zx\" (UID: \"732d9d71-d680-4291-aaee-028307ef79dc\") " pod="kube-system/global-pull-secret-syncer-gz9zx"
Sep 09 20:52:24.258322 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.257386    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/dfe1a979-9dff-419b-b517-93e03c19c040-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9rprp\" (UID: \"dfe1a979-9dff-419b-b517-93e03c19c040\") " pod="openshift-multus/multus-additional-cni-plugins-9rprp"
Sep 09 20:52:24.258322 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.257423    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b2481c54-52e5-44b3-a24a-09b4dd9f575b-host-slash\") pod \"ovnkube-node-9dcqf\" (UID: \"b2481c54-52e5-44b3-a24a-09b4dd9f575b\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dcqf"
Sep 09 20:52:24.258322 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.257472    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a55d0249-76a1-42d0-8e06-8c4fa1e557f8-sys-fs\") pod \"aws-ebs-csi-driver-node-pdpk7\" (UID: \"a55d0249-76a1-42d0-8e06-8c4fa1e557f8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pdpk7"
Sep 09 20:52:24.258322 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.257491    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/22f66b47-36ad-4226-b0f9-ad4075eaaa2d-hostroot\") pod \"multus-jbs9t\" (UID: \"22f66b47-36ad-4226-b0f9-ad4075eaaa2d\") " pod="openshift-multus/multus-jbs9t"
Sep 09 20:52:24.258322 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.257506    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/22f66b47-36ad-4226-b0f9-ad4075eaaa2d-etc-kubernetes\") pod \"multus-jbs9t\" (UID: \"22f66b47-36ad-4226-b0f9-ad4075eaaa2d\") " pod="openshift-multus/multus-jbs9t"
Sep 09 20:52:24.258322 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.257521    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b2481c54-52e5-44b3-a24a-09b4dd9f575b-host-cni-bin\") pod \"ovnkube-node-9dcqf\" (UID: \"b2481c54-52e5-44b3-a24a-09b4dd9f575b\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dcqf"
Sep 09 20:52:24.258322 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.257541    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/22f66b47-36ad-4226-b0f9-ad4075eaaa2d-system-cni-dir\") pod \"multus-jbs9t\" (UID: \"22f66b47-36ad-4226-b0f9-ad4075eaaa2d\") " pod="openshift-multus/multus-jbs9t"
Sep 09 20:52:24.258322 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.257567    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6c58d\" (UniqueName: \"kubernetes.io/projected/e1193e63-dea9-41f4-a6ab-748d8ae109e8-kube-api-access-6c58d\") pod \"network-check-target-b6qr7\" (UID: \"e1193e63-dea9-41f4-a6ab-748d8ae109e8\") " pod="openshift-network-diagnostics/network-check-target-b6qr7"
Sep 09 20:52:24.258322 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.257572    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/22f66b47-36ad-4226-b0f9-ad4075eaaa2d-hostroot\") pod \"multus-jbs9t\" (UID: \"22f66b47-36ad-4226-b0f9-ad4075eaaa2d\") " pod="openshift-multus/multus-jbs9t"
Sep 09 20:52:24.258322 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.257593    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b2481c54-52e5-44b3-a24a-09b4dd9f575b-env-overrides\") pod \"ovnkube-node-9dcqf\" (UID: \"b2481c54-52e5-44b3-a24a-09b4dd9f575b\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dcqf"
Sep 09 20:52:24.258322 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.257573    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/22f66b47-36ad-4226-b0f9-ad4075eaaa2d-etc-kubernetes\") pod \"multus-jbs9t\" (UID: \"22f66b47-36ad-4226-b0f9-ad4075eaaa2d\") " pod="openshift-multus/multus-jbs9t"
Sep 09 20:52:24.258322 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.257617    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a55d0249-76a1-42d0-8e06-8c4fa1e557f8-socket-dir\") pod \"aws-ebs-csi-driver-node-pdpk7\" (UID: \"a55d0249-76a1-42d0-8e06-8c4fa1e557f8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pdpk7"
Sep 09 20:52:24.258322 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.257652    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/10f78cd8-5259-4024-ae5f-68bfa6a0626f-tmp\") pod \"tuned-s98q4\" (UID: \"10f78cd8-5259-4024-ae5f-68bfa6a0626f\") " pod="openshift-cluster-node-tuning-operator/tuned-s98q4"
Sep 09 20:52:24.258322 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.257689    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dfe1a979-9dff-419b-b517-93e03c19c040-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9rprp\" (UID: \"dfe1a979-9dff-419b-b517-93e03c19c040\") " pod="openshift-multus/multus-additional-cni-plugins-9rprp"
Sep 09 20:52:24.258322 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.257719    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ggfx\" (UniqueName: \"kubernetes.io/projected/dfe1a979-9dff-419b-b517-93e03c19c040-kube-api-access-9ggfx\") pod \"multus-additional-cni-plugins-9rprp\" (UID: \"dfe1a979-9dff-419b-b517-93e03c19c040\") " pod="openshift-multus/multus-additional-cni-plugins-9rprp"
Sep 09 20:52:24.258322 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.257718    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/22f66b47-36ad-4226-b0f9-ad4075eaaa2d-system-cni-dir\") pod \"multus-jbs9t\" (UID: \"22f66b47-36ad-4226-b0f9-ad4075eaaa2d\") " pod="openshift-multus/multus-jbs9t"
Sep 09 20:52:24.259115 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.257812    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dfe1a979-9dff-419b-b517-93e03c19c040-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9rprp\" (UID: \"dfe1a979-9dff-419b-b517-93e03c19c040\") " pod="openshift-multus/multus-additional-cni-plugins-9rprp"
Sep 09 20:52:24.259115 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.257844    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ddb26243-140d-4ac9-8a3d-4079559bd0e7-hosts-file\") pod \"node-resolver-88pfs\" (UID: \"ddb26243-140d-4ac9-8a3d-4079559bd0e7\") " pod="openshift-dns/node-resolver-88pfs"
Sep 09 20:52:24.259115 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.257850    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/22f66b47-36ad-4226-b0f9-ad4075eaaa2d-cni-binary-copy\") pod \"multus-jbs9t\" (UID: \"22f66b47-36ad-4226-b0f9-ad4075eaaa2d\") " pod="openshift-multus/multus-jbs9t"
Sep 09 20:52:24.259115 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.257866    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/732d9d71-d680-4291-aaee-028307ef79dc-dbus\") pod \"global-pull-secret-syncer-gz9zx\" (UID: \"732d9d71-d680-4291-aaee-028307ef79dc\") " pod="kube-system/global-pull-secret-syncer-gz9zx"
Sep 09 20:52:24.259115 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.257870    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/dfe1a979-9dff-419b-b517-93e03c19c040-cni-binary-copy\") pod \"multus-additional-cni-plugins-9rprp\" (UID: \"dfe1a979-9dff-419b-b517-93e03c19c040\") " pod="openshift-multus/multus-additional-cni-plugins-9rprp"
Sep 09 20:52:24.259115 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.257881    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b2481c54-52e5-44b3-a24a-09b4dd9f575b-node-log\") pod \"ovnkube-node-9dcqf\" (UID: \"b2481c54-52e5-44b3-a24a-09b4dd9f575b\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dcqf"
Sep 09 20:52:24.259115 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.257901    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ddb26243-140d-4ac9-8a3d-4079559bd0e7-hosts-file\") pod \"node-resolver-88pfs\" (UID: \"ddb26243-140d-4ac9-8a3d-4079559bd0e7\") " pod="openshift-dns/node-resolver-88pfs"
Sep 09 20:52:24.259115 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.257903    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a55d0249-76a1-42d0-8e06-8c4fa1e557f8-device-dir\") pod \"aws-ebs-csi-driver-node-pdpk7\" (UID: \"a55d0249-76a1-42d0-8e06-8c4fa1e557f8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pdpk7"
Sep 09 20:52:24.259115 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.257928    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/10f78cd8-5259-4024-ae5f-68bfa6a0626f-etc-modprobe-d\") pod \"tuned-s98q4\" (UID: \"10f78cd8-5259-4024-ae5f-68bfa6a0626f\") " pod="openshift-cluster-node-tuning-operator/tuned-s98q4"
Sep 09 20:52:24.259115 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.257945    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/22f66b47-36ad-4226-b0f9-ad4075eaaa2d-host-var-lib-kubelet\") pod \"multus-jbs9t\" (UID: \"22f66b47-36ad-4226-b0f9-ad4075eaaa2d\") " pod="openshift-multus/multus-jbs9t"
Sep 09 20:52:24.259115 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.257961    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ddb26243-140d-4ac9-8a3d-4079559bd0e7-tmp-dir\") pod \"node-resolver-88pfs\" (UID: \"ddb26243-140d-4ac9-8a3d-4079559bd0e7\") " pod="openshift-dns/node-resolver-88pfs"
Sep 09 20:52:24.259115 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.257999    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/22f66b47-36ad-4226-b0f9-ad4075eaaa2d-host-var-lib-kubelet\") pod \"multus-jbs9t\" (UID: \"22f66b47-36ad-4226-b0f9-ad4075eaaa2d\") " pod="openshift-multus/multus-jbs9t"
Sep 09 20:52:24.259115 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.258033    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5r9bj\" (UniqueName: \"kubernetes.io/projected/169cade8-7395-4b62-9544-ff0c47022a89-kube-api-access-5r9bj\") pod \"node-ca-tml69\" (UID: \"169cade8-7395-4b62-9544-ff0c47022a89\") " pod="openshift-image-registry/node-ca-tml69"
Sep 09 20:52:24.259115 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.258077    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdgb2\" (UniqueName: \"kubernetes.io/projected/b2481c54-52e5-44b3-a24a-09b4dd9f575b-kube-api-access-bdgb2\") pod \"ovnkube-node-9dcqf\" (UID: \"b2481c54-52e5-44b3-a24a-09b4dd9f575b\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dcqf"
Sep 09 20:52:24.259115 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.258131    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/28da7add-257c-49f6-a846-6e8116b68d4c-iptables-alerter-script\") pod \"iptables-alerter-kfpbw\" (UID: \"28da7add-257c-49f6-a846-6e8116b68d4c\") " pod="openshift-network-operator/iptables-alerter-kfpbw"
Sep 09 20:52:24.259115 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.258158    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b2481c54-52e5-44b3-a24a-09b4dd9f575b-ovn-node-metrics-cert\") pod \"ovnkube-node-9dcqf\" (UID: \"b2481c54-52e5-44b3-a24a-09b4dd9f575b\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dcqf"
Sep 09 20:52:24.259115 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.258180    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ddb26243-140d-4ac9-8a3d-4079559bd0e7-tmp-dir\") pod \"node-resolver-88pfs\" (UID: \"ddb26243-140d-4ac9-8a3d-4079559bd0e7\") " pod="openshift-dns/node-resolver-88pfs"
Sep 09 20:52:24.259772 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.258188    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/dfe1a979-9dff-419b-b517-93e03c19c040-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9rprp\" (UID: \"dfe1a979-9dff-419b-b517-93e03c19c040\") " pod="openshift-multus/multus-additional-cni-plugins-9rprp"
Sep 09 20:52:24.259772 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.258217    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/22f66b47-36ad-4226-b0f9-ad4075eaaa2d-multus-cni-dir\") pod \"multus-jbs9t\" (UID: \"22f66b47-36ad-4226-b0f9-ad4075eaaa2d\") " pod="openshift-multus/multus-jbs9t"
Sep 09 20:52:24.259772 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.258242    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b2481c54-52e5-44b3-a24a-09b4dd9f575b-var-lib-openvswitch\") pod \"ovnkube-node-9dcqf\" (UID: \"b2481c54-52e5-44b3-a24a-09b4dd9f575b\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dcqf"
Sep 09 20:52:24.259772 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.258265    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b2481c54-52e5-44b3-a24a-09b4dd9f575b-run-ovn\") pod \"ovnkube-node-9dcqf\" (UID: \"b2481c54-52e5-44b3-a24a-09b4dd9f575b\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dcqf"
Sep 09 20:52:24.259772 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.258303    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/22f66b47-36ad-4226-b0f9-ad4075eaaa2d-multus-cni-dir\") pod \"multus-jbs9t\" (UID: \"22f66b47-36ad-4226-b0f9-ad4075eaaa2d\") " pod="openshift-multus/multus-jbs9t"
Sep 09 20:52:24.259772 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.258331    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crvp7\" (UniqueName: \"kubernetes.io/projected/c6fd107c-ea2f-4e85-9a32-1e11f0858fd3-kube-api-access-crvp7\") pod \"network-metrics-daemon-ws8hc\" (UID: \"c6fd107c-ea2f-4e85-9a32-1e11f0858fd3\") " pod="openshift-multus/network-metrics-daemon-ws8hc"
Sep 09 20:52:24.259772 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.258365    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b2481c54-52e5-44b3-a24a-09b4dd9f575b-log-socket\") pod \"ovnkube-node-9dcqf\" (UID: \"b2481c54-52e5-44b3-a24a-09b4dd9f575b\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dcqf"
Sep 09 20:52:24.259772 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.258388    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b2481c54-52e5-44b3-a24a-09b4dd9f575b-host-cni-netd\") pod \"ovnkube-node-9dcqf\" (UID: \"b2481c54-52e5-44b3-a24a-09b4dd9f575b\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dcqf"
Sep 09 20:52:24.259772 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.258424    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dfe1a979-9dff-419b-b517-93e03c19c040-system-cni-dir\") pod \"multus-additional-cni-plugins-9rprp\" (UID: \"dfe1a979-9dff-419b-b517-93e03c19c040\") " pod="openshift-multus/multus-additional-cni-plugins-9rprp"
Sep 09 20:52:24.259772 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.258484    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/22f66b47-36ad-4226-b0f9-ad4075eaaa2d-host-var-lib-cni-multus\") pod \"multus-jbs9t\" (UID: \"22f66b47-36ad-4226-b0f9-ad4075eaaa2d\") " pod="openshift-multus/multus-jbs9t"
Sep 09 20:52:24.259772 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.258509    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/22f66b47-36ad-4226-b0f9-ad4075eaaa2d-multus-daemon-config\") pod \"multus-jbs9t\" (UID: \"22f66b47-36ad-4226-b0f9-ad4075eaaa2d\") " pod="openshift-multus/multus-jbs9t"
Sep 09 20:52:24.259772 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.258487    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dfe1a979-9dff-419b-b517-93e03c19c040-system-cni-dir\") pod \"multus-additional-cni-plugins-9rprp\" (UID: \"dfe1a979-9dff-419b-b517-93e03c19c040\") " pod="openshift-multus/multus-additional-cni-plugins-9rprp"
Sep 09 20:52:24.259772 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.258534    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/169cade8-7395-4b62-9544-ff0c47022a89-serviceca\") pod \"node-ca-tml69\" (UID: \"169cade8-7395-4b62-9544-ff0c47022a89\") " pod="openshift-image-registry/node-ca-tml69"
Sep 09 20:52:24.259772 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.258571    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/22f66b47-36ad-4226-b0f9-ad4075eaaa2d-host-var-lib-cni-multus\") pod \"multus-jbs9t\" (UID: \"22f66b47-36ad-4226-b0f9-ad4075eaaa2d\") " pod="openshift-multus/multus-jbs9t"
Sep 09 20:52:24.259772 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.258599    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b2481c54-52e5-44b3-a24a-09b4dd9f575b-host-run-netns\") pod \"ovnkube-node-9dcqf\" (UID: \"b2481c54-52e5-44b3-a24a-09b4dd9f575b\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dcqf"
Sep 09 20:52:24.259772 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.258655    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/dfe1a979-9dff-419b-b517-93e03c19c040-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9rprp\" (UID: \"dfe1a979-9dff-419b-b517-93e03c19c040\") " pod="openshift-multus/multus-additional-cni-plugins-9rprp"
Sep 09 20:52:24.259772 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.258947    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/22f66b47-36ad-4226-b0f9-ad4075eaaa2d-multus-daemon-config\") pod \"multus-jbs9t\" (UID: \"22f66b47-36ad-4226-b0f9-ad4075eaaa2d\") " pod="openshift-multus/multus-jbs9t"
Sep 09 20:52:24.275024 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.274996    2579 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 09 20:52:24.278835 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.278810    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-prbqs\" (UniqueName: \"kubernetes.io/projected/22f66b47-36ad-4226-b0f9-ad4075eaaa2d-kube-api-access-prbqs\") pod \"multus-jbs9t\" (UID: \"22f66b47-36ad-4226-b0f9-ad4075eaaa2d\") " pod="openshift-multus/multus-jbs9t"
Sep 09 20:52:24.278919 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.278819    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ggfx\" (UniqueName: \"kubernetes.io/projected/dfe1a979-9dff-419b-b517-93e03c19c040-kube-api-access-9ggfx\") pod \"multus-additional-cni-plugins-9rprp\" (UID: \"dfe1a979-9dff-419b-b517-93e03c19c040\") " pod="openshift-multus/multus-additional-cni-plugins-9rprp"
Sep 09 20:52:24.278919 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.278858    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fq6gl\" (UniqueName: \"kubernetes.io/projected/ddb26243-140d-4ac9-8a3d-4079559bd0e7-kube-api-access-fq6gl\") pod \"node-resolver-88pfs\" (UID: \"ddb26243-140d-4ac9-8a3d-4079559bd0e7\") " pod="openshift-dns/node-resolver-88pfs"
Sep 09 20:52:24.358850 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.358821    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a55d0249-76a1-42d0-8e06-8c4fa1e557f8-registration-dir\") pod \"aws-ebs-csi-driver-node-pdpk7\" (UID: \"a55d0249-76a1-42d0-8e06-8c4fa1e557f8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pdpk7"
Sep 09 20:52:24.358850 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.358849    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnkxj\" (UniqueName: \"kubernetes.io/projected/28da7add-257c-49f6-a846-6e8116b68d4c-kube-api-access-tnkxj\") pod \"iptables-alerter-kfpbw\" (UID: \"28da7add-257c-49f6-a846-6e8116b68d4c\") " pod="openshift-network-operator/iptables-alerter-kfpbw"
Sep 09 20:52:24.358992 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.358865    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9075ffbd-9633-49ac-b6aa-f410776c4ad6-konnectivity-ca\") pod \"konnectivity-agent-kwvw7\" (UID: \"9075ffbd-9633-49ac-b6aa-f410776c4ad6\") " pod="kube-system/konnectivity-agent-kwvw7"
Sep 09 20:52:24.358992 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.358876    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a55d0249-76a1-42d0-8e06-8c4fa1e557f8-registration-dir\") pod \"aws-ebs-csi-driver-node-pdpk7\" (UID: \"a55d0249-76a1-42d0-8e06-8c4fa1e557f8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pdpk7"
Sep 09 20:52:24.358992 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.358881    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c6fd107c-ea2f-4e85-9a32-1e11f0858fd3-metrics-certs\") pod \"network-metrics-daemon-ws8hc\" (UID: \"c6fd107c-ea2f-4e85-9a32-1e11f0858fd3\") " pod="openshift-multus/network-metrics-daemon-ws8hc"
Sep 09 20:52:24.358992 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.358916    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b2481c54-52e5-44b3-a24a-09b4dd9f575b-run-openvswitch\") pod \"ovnkube-node-9dcqf\" (UID: \"b2481c54-52e5-44b3-a24a-09b4dd9f575b\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dcqf"
Sep 09 20:52:24.358992 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.358931    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a55d0249-76a1-42d0-8e06-8c4fa1e557f8-kubelet-dir\") pod \"aws-ebs-csi-driver-node-pdpk7\" (UID: \"a55d0249-76a1-42d0-8e06-8c4fa1e557f8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pdpk7"
Sep 09 20:52:24.358992 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.358945    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/10f78cd8-5259-4024-ae5f-68bfa6a0626f-etc-tuned\") pod \"tuned-s98q4\" (UID: \"10f78cd8-5259-4024-ae5f-68bfa6a0626f\") " pod="openshift-cluster-node-tuning-operator/tuned-s98q4"
Sep 09 20:52:24.358992 ip-10-0-138-224 kubenswrapper[2579]: E0909 20:52:24.358958    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:52:24.359271 ip-10-0-138-224 kubenswrapper[2579]: E0909 20:52:24.359013    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c6fd107c-ea2f-4e85-9a32-1e11f0858fd3-metrics-certs podName:c6fd107c-ea2f-4e85-9a32-1e11f0858fd3 nodeName:}" failed. No retries permitted until 2026-09-09 20:52:24.858998403 +0000 UTC m=+2.099092498 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c6fd107c-ea2f-4e85-9a32-1e11f0858fd3-metrics-certs") pod "network-metrics-daemon-ws8hc" (UID: "c6fd107c-ea2f-4e85-9a32-1e11f0858fd3") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:52:24.359271 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.359030    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/10f78cd8-5259-4024-ae5f-68bfa6a0626f-etc-sysctl-d\") pod \"tuned-s98q4\" (UID: \"10f78cd8-5259-4024-ae5f-68bfa6a0626f\") " pod="openshift-cluster-node-tuning-operator/tuned-s98q4"
Sep 09 20:52:24.359271 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.359064    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b2481c54-52e5-44b3-a24a-09b4dd9f575b-run-openvswitch\") pod \"ovnkube-node-9dcqf\" (UID: \"b2481c54-52e5-44b3-a24a-09b4dd9f575b\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dcqf"
Sep 09 20:52:24.359271 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.358960    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/10f78cd8-5259-4024-ae5f-68bfa6a0626f-etc-sysctl-d\") pod \"tuned-s98q4\" (UID: \"10f78cd8-5259-4024-ae5f-68bfa6a0626f\") " pod="openshift-cluster-node-tuning-operator/tuned-s98q4"
Sep 09 20:52:24.359271 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.359123    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b2481c54-52e5-44b3-a24a-09b4dd9f575b-host-kubelet\") pod \"ovnkube-node-9dcqf\" (UID: \"b2481c54-52e5-44b3-a24a-09b4dd9f575b\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dcqf"
Sep 09 20:52:24.359271 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.359163    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b2481c54-52e5-44b3-a24a-09b4dd9f575b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-9dcqf\" (UID: \"b2481c54-52e5-44b3-a24a-09b4dd9f575b\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dcqf"
Sep 09 20:52:24.359271 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.359168    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b2481c54-52e5-44b3-a24a-09b4dd9f575b-host-kubelet\") pod \"ovnkube-node-9dcqf\" (UID: \"b2481c54-52e5-44b3-a24a-09b4dd9f575b\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dcqf"
Sep 09 20:52:24.359271 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.359129    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a55d0249-76a1-42d0-8e06-8c4fa1e557f8-kubelet-dir\") pod \"aws-ebs-csi-driver-node-pdpk7\" (UID: \"a55d0249-76a1-42d0-8e06-8c4fa1e557f8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pdpk7"
Sep 09 20:52:24.359271 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.359190    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b2481c54-52e5-44b3-a24a-09b4dd9f575b-ovnkube-script-lib\") pod \"ovnkube-node-9dcqf\" (UID: \"b2481c54-52e5-44b3-a24a-09b4dd9f575b\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dcqf"
Sep 09 20:52:24.359271 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.359217    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a55d0249-76a1-42d0-8e06-8c4fa1e557f8-etc-selinux\") pod \"aws-ebs-csi-driver-node-pdpk7\" (UID: \"a55d0249-76a1-42d0-8e06-8c4fa1e557f8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pdpk7"
Sep 09 20:52:24.359271 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.359241    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/10f78cd8-5259-4024-ae5f-68bfa6a0626f-etc-systemd\") pod \"tuned-s98q4\" (UID: \"10f78cd8-5259-4024-ae5f-68bfa6a0626f\") " pod="openshift-cluster-node-tuning-operator/tuned-s98q4"
Sep 09 20:52:24.359271 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.359257    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b2481c54-52e5-44b3-a24a-09b4dd9f575b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-9dcqf\" (UID: \"b2481c54-52e5-44b3-a24a-09b4dd9f575b\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dcqf"
Sep 09 20:52:24.359271 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.359265    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/10f78cd8-5259-4024-ae5f-68bfa6a0626f-lib-modules\") pod \"tuned-s98q4\" (UID: \"10f78cd8-5259-4024-ae5f-68bfa6a0626f\") " pod="openshift-cluster-node-tuning-operator/tuned-s98q4"
Sep 09 20:52:24.359809 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.359291    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a55d0249-76a1-42d0-8e06-8c4fa1e557f8-etc-selinux\") pod \"aws-ebs-csi-driver-node-pdpk7\" (UID: \"a55d0249-76a1-42d0-8e06-8c4fa1e557f8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pdpk7"
Sep 09 20:52:24.359809 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.359308    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b2481c54-52e5-44b3-a24a-09b4dd9f575b-host-run-ovn-kubernetes\") pod \"ovnkube-node-9dcqf\" (UID: \"b2481c54-52e5-44b3-a24a-09b4dd9f575b\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dcqf"
Sep 09 20:52:24.359809 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.359320    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/10f78cd8-5259-4024-ae5f-68bfa6a0626f-etc-systemd\") pod \"tuned-s98q4\" (UID: \"10f78cd8-5259-4024-ae5f-68bfa6a0626f\") " pod="openshift-cluster-node-tuning-operator/tuned-s98q4"
Sep 09 20:52:24.359809 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.359354    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/732d9d71-d680-4291-aaee-028307ef79dc-kubelet-config\") pod \"global-pull-secret-syncer-gz9zx\" (UID: \"732d9d71-d680-4291-aaee-028307ef79dc\") " pod="kube-system/global-pull-secret-syncer-gz9zx"
Sep 09 20:52:24.359809 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.359369    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b2481c54-52e5-44b3-a24a-09b4dd9f575b-host-run-ovn-kubernetes\") pod \"ovnkube-node-9dcqf\" (UID: \"b2481c54-52e5-44b3-a24a-09b4dd9f575b\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dcqf"
Sep 09 20:52:24.359809 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.359379    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/169cade8-7395-4b62-9544-ff0c47022a89-host\") pod \"node-ca-tml69\" (UID: \"169cade8-7395-4b62-9544-ff0c47022a89\") " pod="openshift-image-registry/node-ca-tml69"
Sep 09 20:52:24.359809 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.359421    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/732d9d71-d680-4291-aaee-028307ef79dc-kubelet-config\") pod \"global-pull-secret-syncer-gz9zx\" (UID: \"732d9d71-d680-4291-aaee-028307ef79dc\") " pod="kube-system/global-pull-secret-syncer-gz9zx"
Sep 09 20:52:24.359809 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.359456    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/10f78cd8-5259-4024-ae5f-68bfa6a0626f-lib-modules\") pod \"tuned-s98q4\" (UID: \"10f78cd8-5259-4024-ae5f-68bfa6a0626f\") " pod="openshift-cluster-node-tuning-operator/tuned-s98q4"
Sep 09 20:52:24.359809 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.359465    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b2481c54-52e5-44b3-a24a-09b4dd9f575b-systemd-units\") pod \"ovnkube-node-9dcqf\" (UID: \"b2481c54-52e5-44b3-a24a-09b4dd9f575b\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dcqf"
Sep 09 20:52:24.359809 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.359462    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/169cade8-7395-4b62-9544-ff0c47022a89-host\") pod \"node-ca-tml69\" (UID: \"169cade8-7395-4b62-9544-ff0c47022a89\") " pod="openshift-image-registry/node-ca-tml69"
Sep 09 20:52:24.359809 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.359511    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/10f78cd8-5259-4024-ae5f-68bfa6a0626f-etc-sysctl-conf\") pod \"tuned-s98q4\" (UID: \"10f78cd8-5259-4024-ae5f-68bfa6a0626f\") " pod="openshift-cluster-node-tuning-operator/tuned-s98q4"
Sep 09 20:52:24.359809 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.359518    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b2481c54-52e5-44b3-a24a-09b4dd9f575b-systemd-units\") pod \"ovnkube-node-9dcqf\" (UID: \"b2481c54-52e5-44b3-a24a-09b4dd9f575b\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dcqf"
Sep 09 20:52:24.359809 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.359536    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/10f78cd8-5259-4024-ae5f-68bfa6a0626f-var-lib-kubelet\") pod \"tuned-s98q4\" (UID: \"10f78cd8-5259-4024-ae5f-68bfa6a0626f\") " pod="openshift-cluster-node-tuning-operator/tuned-s98q4"
Sep 09 20:52:24.359809 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.359549    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9075ffbd-9633-49ac-b6aa-f410776c4ad6-konnectivity-ca\") pod \"konnectivity-agent-kwvw7\" (UID: \"9075ffbd-9633-49ac-b6aa-f410776c4ad6\") " pod="kube-system/konnectivity-agent-kwvw7"
Sep 09 20:52:24.359809 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.359560    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/10f78cd8-5259-4024-ae5f-68bfa6a0626f-host\") pod \"tuned-s98q4\" (UID: \"10f78cd8-5259-4024-ae5f-68bfa6a0626f\") " pod="openshift-cluster-node-tuning-operator/tuned-s98q4"
Sep 09 20:52:24.359809 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.359587    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9075ffbd-9633-49ac-b6aa-f410776c4ad6-agent-certs\") pod \"konnectivity-agent-kwvw7\" (UID: \"9075ffbd-9633-49ac-b6aa-f410776c4ad6\") " pod="kube-system/konnectivity-agent-kwvw7"
Sep 09 20:52:24.359809 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.359593    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/10f78cd8-5259-4024-ae5f-68bfa6a0626f-var-lib-kubelet\") pod \"tuned-s98q4\" (UID: \"10f78cd8-5259-4024-ae5f-68bfa6a0626f\") " pod="openshift-cluster-node-tuning-operator/tuned-s98q4"
Sep 09 20:52:24.359809 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.359629    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/10f78cd8-5259-4024-ae5f-68bfa6a0626f-etc-sysctl-conf\") pod \"tuned-s98q4\" (UID: \"10f78cd8-5259-4024-ae5f-68bfa6a0626f\") " pod="openshift-cluster-node-tuning-operator/tuned-s98q4"
Sep 09 20:52:24.360523 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.359634    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/10f78cd8-5259-4024-ae5f-68bfa6a0626f-host\") pod \"tuned-s98q4\" (UID: \"10f78cd8-5259-4024-ae5f-68bfa6a0626f\") " pod="openshift-cluster-node-tuning-operator/tuned-s98q4"
Sep 09 20:52:24.360523 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.359622    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b2481c54-52e5-44b3-a24a-09b4dd9f575b-run-systemd\") pod \"ovnkube-node-9dcqf\" (UID: \"b2481c54-52e5-44b3-a24a-09b4dd9f575b\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dcqf"
Sep 09 20:52:24.360523 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.359720    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b2481c54-52e5-44b3-a24a-09b4dd9f575b-run-systemd\") pod \"ovnkube-node-9dcqf\" (UID: \"b2481c54-52e5-44b3-a24a-09b4dd9f575b\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dcqf"
Sep 09 20:52:24.360523 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.359801    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b2481c54-52e5-44b3-a24a-09b4dd9f575b-ovnkube-script-lib\") pod \"ovnkube-node-9dcqf\" (UID: \"b2481c54-52e5-44b3-a24a-09b4dd9f575b\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dcqf"
Sep 09 20:52:24.360523 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.359803    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b2481c54-52e5-44b3-a24a-09b4dd9f575b-etc-openvswitch\") pod \"ovnkube-node-9dcqf\" (UID: \"b2481c54-52e5-44b3-a24a-09b4dd9f575b\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dcqf"
Sep 09 20:52:24.360523 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.359840    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/10f78cd8-5259-4024-ae5f-68bfa6a0626f-etc-sysconfig\") pod \"tuned-s98q4\" (UID: \"10f78cd8-5259-4024-ae5f-68bfa6a0626f\") " pod="openshift-cluster-node-tuning-operator/tuned-s98q4"
Sep 09 20:52:24.360523 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.359840    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b2481c54-52e5-44b3-a24a-09b4dd9f575b-etc-openvswitch\") pod \"ovnkube-node-9dcqf\" (UID: \"b2481c54-52e5-44b3-a24a-09b4dd9f575b\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dcqf"
Sep 09 20:52:24.360523 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.359867    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/10f78cd8-5259-4024-ae5f-68bfa6a0626f-etc-kubernetes\") pod \"tuned-s98q4\" (UID: \"10f78cd8-5259-4024-ae5f-68bfa6a0626f\") " pod="openshift-cluster-node-tuning-operator/tuned-s98q4"
Sep 09 20:52:24.360523 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.359884    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmgq4\" (UniqueName: \"kubernetes.io/projected/10f78cd8-5259-4024-ae5f-68bfa6a0626f-kube-api-access-gmgq4\") pod \"tuned-s98q4\" (UID: \"10f78cd8-5259-4024-ae5f-68bfa6a0626f\") " pod="openshift-cluster-node-tuning-operator/tuned-s98q4"
Sep 09 20:52:24.360523 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.359898    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/10f78cd8-5259-4024-ae5f-68bfa6a0626f-etc-sysconfig\") pod \"tuned-s98q4\" (UID: \"10f78cd8-5259-4024-ae5f-68bfa6a0626f\") " pod="openshift-cluster-node-tuning-operator/tuned-s98q4"
Sep 09 20:52:24.360523 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.359901    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwfng\" (UniqueName: \"kubernetes.io/projected/a55d0249-76a1-42d0-8e06-8c4fa1e557f8-kube-api-access-pwfng\") pod \"aws-ebs-csi-driver-node-pdpk7\" (UID: \"a55d0249-76a1-42d0-8e06-8c4fa1e557f8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pdpk7"
Sep 09 20:52:24.360523 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.360064    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/10f78cd8-5259-4024-ae5f-68bfa6a0626f-run\") pod \"tuned-s98q4\" (UID: \"10f78cd8-5259-4024-ae5f-68bfa6a0626f\") " pod="openshift-cluster-node-tuning-operator/tuned-s98q4"
Sep 09 20:52:24.360523 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.360083    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/732d9d71-d680-4291-aaee-028307ef79dc-original-pull-secret\") pod \"global-pull-secret-syncer-gz9zx\" (UID: \"732d9d71-d680-4291-aaee-028307ef79dc\") " pod="kube-system/global-pull-secret-syncer-gz9zx"
Sep 09 20:52:24.360523 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.360100    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b2481c54-52e5-44b3-a24a-09b4dd9f575b-host-slash\") pod \"ovnkube-node-9dcqf\" (UID: \"b2481c54-52e5-44b3-a24a-09b4dd9f575b\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dcqf"
Sep 09 20:52:24.360523 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.360120    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a55d0249-76a1-42d0-8e06-8c4fa1e557f8-sys-fs\") pod \"aws-ebs-csi-driver-node-pdpk7\" (UID: \"a55d0249-76a1-42d0-8e06-8c4fa1e557f8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pdpk7"
Sep 09 20:52:24.360523 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.360137    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b2481c54-52e5-44b3-a24a-09b4dd9f575b-host-cni-bin\") pod \"ovnkube-node-9dcqf\" (UID: \"b2481c54-52e5-44b3-a24a-09b4dd9f575b\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dcqf"
Sep 09 20:52:24.360523 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.360155    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6c58d\" (UniqueName: \"kubernetes.io/projected/e1193e63-dea9-41f4-a6ab-748d8ae109e8-kube-api-access-6c58d\") pod \"network-check-target-b6qr7\" (UID: \"e1193e63-dea9-41f4-a6ab-748d8ae109e8\") " pod="openshift-network-diagnostics/network-check-target-b6qr7"
Sep 09 20:52:24.362228 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.360192    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/10f78cd8-5259-4024-ae5f-68bfa6a0626f-etc-kubernetes\") pod \"tuned-s98q4\" (UID: \"10f78cd8-5259-4024-ae5f-68bfa6a0626f\") " pod="openshift-cluster-node-tuning-operator/tuned-s98q4"
Sep 09 20:52:24.362228 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.360197    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b2481c54-52e5-44b3-a24a-09b4dd9f575b-host-slash\") pod \"ovnkube-node-9dcqf\" (UID: \"b2481c54-52e5-44b3-a24a-09b4dd9f575b\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dcqf"
Sep 09 20:52:24.362228 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.360208    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b2481c54-52e5-44b3-a24a-09b4dd9f575b-env-overrides\") pod \"ovnkube-node-9dcqf\" (UID: \"b2481c54-52e5-44b3-a24a-09b4dd9f575b\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dcqf"
Sep 09 20:52:24.362228 ip-10-0-138-224 kubenswrapper[2579]: E0909 20:52:24.360218    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 20:52:24.362228 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.360235    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a55d0249-76a1-42d0-8e06-8c4fa1e557f8-socket-dir\") pod \"aws-ebs-csi-driver-node-pdpk7\" (UID: \"a55d0249-76a1-42d0-8e06-8c4fa1e557f8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pdpk7"
Sep 09 20:52:24.362228 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.360229    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b2481c54-52e5-44b3-a24a-09b4dd9f575b-host-cni-bin\") pod \"ovnkube-node-9dcqf\" (UID: \"b2481c54-52e5-44b3-a24a-09b4dd9f575b\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dcqf"
Sep 09 20:52:24.362228 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.360257    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/10f78cd8-5259-4024-ae5f-68bfa6a0626f-tmp\") pod \"tuned-s98q4\" (UID: \"10f78cd8-5259-4024-ae5f-68bfa6a0626f\") " pod="openshift-cluster-node-tuning-operator/tuned-s98q4"
Sep 09 20:52:24.362228 ip-10-0-138-224 kubenswrapper[2579]: E0909 20:52:24.360287    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/732d9d71-d680-4291-aaee-028307ef79dc-original-pull-secret podName:732d9d71-d680-4291-aaee-028307ef79dc nodeName:}" failed. No retries permitted until 2026-09-09 20:52:24.860269829 +0000 UTC m=+2.100363965 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/732d9d71-d680-4291-aaee-028307ef79dc-original-pull-secret") pod "global-pull-secret-syncer-gz9zx" (UID: "732d9d71-d680-4291-aaee-028307ef79dc") : object "kube-system"/"original-pull-secret" not registered
Sep 09 20:52:24.362228 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.360302    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/10f78cd8-5259-4024-ae5f-68bfa6a0626f-run\") pod \"tuned-s98q4\" (UID: \"10f78cd8-5259-4024-ae5f-68bfa6a0626f\") " pod="openshift-cluster-node-tuning-operator/tuned-s98q4"
Sep 09 20:52:24.362228 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.360329    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/732d9d71-d680-4291-aaee-028307ef79dc-dbus\") pod \"global-pull-secret-syncer-gz9zx\" (UID: \"732d9d71-d680-4291-aaee-028307ef79dc\") " pod="kube-system/global-pull-secret-syncer-gz9zx"
Sep 09 20:52:24.362228 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.360354    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a55d0249-76a1-42d0-8e06-8c4fa1e557f8-sys-fs\") pod \"aws-ebs-csi-driver-node-pdpk7\" (UID: \"a55d0249-76a1-42d0-8e06-8c4fa1e557f8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pdpk7"
Sep 09 20:52:24.362228 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.360357    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b2481c54-52e5-44b3-a24a-09b4dd9f575b-node-log\") pod \"ovnkube-node-9dcqf\" (UID: \"b2481c54-52e5-44b3-a24a-09b4dd9f575b\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dcqf"
Sep 09 20:52:24.362228 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.360394    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a55d0249-76a1-42d0-8e06-8c4fa1e557f8-device-dir\") pod \"aws-ebs-csi-driver-node-pdpk7\" (UID: \"a55d0249-76a1-42d0-8e06-8c4fa1e557f8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pdpk7"
Sep 09 20:52:24.362228 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.360407    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/732d9d71-d680-4291-aaee-028307ef79dc-dbus\") pod \"global-pull-secret-syncer-gz9zx\" (UID: \"732d9d71-d680-4291-aaee-028307ef79dc\") " pod="kube-system/global-pull-secret-syncer-gz9zx"
Sep 09 20:52:24.362228 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.360417    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/10f78cd8-5259-4024-ae5f-68bfa6a0626f-etc-modprobe-d\") pod \"tuned-s98q4\" (UID: \"10f78cd8-5259-4024-ae5f-68bfa6a0626f\") " pod="openshift-cluster-node-tuning-operator/tuned-s98q4"
Sep 09 20:52:24.362228 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.360466    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b2481c54-52e5-44b3-a24a-09b4dd9f575b-node-log\") pod \"ovnkube-node-9dcqf\" (UID: \"b2481c54-52e5-44b3-a24a-09b4dd9f575b\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dcqf"
Sep 09 20:52:24.362228 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.360464    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5r9bj\" (UniqueName: \"kubernetes.io/projected/169cade8-7395-4b62-9544-ff0c47022a89-kube-api-access-5r9bj\") pod \"node-ca-tml69\" (UID: \"169cade8-7395-4b62-9544-ff0c47022a89\") " pod="openshift-image-registry/node-ca-tml69"
Sep 09 20:52:24.362855 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.360509    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdgb2\" (UniqueName: \"kubernetes.io/projected/b2481c54-52e5-44b3-a24a-09b4dd9f575b-kube-api-access-bdgb2\") pod \"ovnkube-node-9dcqf\" (UID: \"b2481c54-52e5-44b3-a24a-09b4dd9f575b\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dcqf"
Sep 09 20:52:24.362855 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.360524    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a55d0249-76a1-42d0-8e06-8c4fa1e557f8-device-dir\") pod \"aws-ebs-csi-driver-node-pdpk7\" (UID: \"a55d0249-76a1-42d0-8e06-8c4fa1e557f8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pdpk7"
Sep 09 20:52:24.362855 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.360532    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/28da7add-257c-49f6-a846-6e8116b68d4c-iptables-alerter-script\") pod \"iptables-alerter-kfpbw\" (UID: \"28da7add-257c-49f6-a846-6e8116b68d4c\") " pod="openshift-network-operator/iptables-alerter-kfpbw"
Sep 09 20:52:24.362855 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.360556    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/10f78cd8-5259-4024-ae5f-68bfa6a0626f-etc-modprobe-d\") pod \"tuned-s98q4\" (UID: \"10f78cd8-5259-4024-ae5f-68bfa6a0626f\") " pod="openshift-cluster-node-tuning-operator/tuned-s98q4"
Sep 09 20:52:24.362855 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.360559    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b2481c54-52e5-44b3-a24a-09b4dd9f575b-ovn-node-metrics-cert\") pod \"ovnkube-node-9dcqf\" (UID: \"b2481c54-52e5-44b3-a24a-09b4dd9f575b\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dcqf"
Sep 09 20:52:24.362855 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.360595    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b2481c54-52e5-44b3-a24a-09b4dd9f575b-var-lib-openvswitch\") pod \"ovnkube-node-9dcqf\" (UID: \"b2481c54-52e5-44b3-a24a-09b4dd9f575b\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dcqf"
Sep 09 20:52:24.362855 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.360619    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b2481c54-52e5-44b3-a24a-09b4dd9f575b-run-ovn\") pod \"ovnkube-node-9dcqf\" (UID: \"b2481c54-52e5-44b3-a24a-09b4dd9f575b\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dcqf"
Sep 09 20:52:24.362855 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.360646    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crvp7\" (UniqueName: \"kubernetes.io/projected/c6fd107c-ea2f-4e85-9a32-1e11f0858fd3-kube-api-access-crvp7\") pod \"network-metrics-daemon-ws8hc\" (UID: \"c6fd107c-ea2f-4e85-9a32-1e11f0858fd3\") " pod="openshift-multus/network-metrics-daemon-ws8hc"
Sep 09 20:52:24.362855 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.360670    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b2481c54-52e5-44b3-a24a-09b4dd9f575b-log-socket\") pod \"ovnkube-node-9dcqf\" (UID: \"b2481c54-52e5-44b3-a24a-09b4dd9f575b\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dcqf"
Sep 09 20:52:24.362855 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.360697    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b2481c54-52e5-44b3-a24a-09b4dd9f575b-env-overrides\") pod \"ovnkube-node-9dcqf\" (UID: \"b2481c54-52e5-44b3-a24a-09b4dd9f575b\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dcqf"
Sep 09 20:52:24.362855 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.360706    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b2481c54-52e5-44b3-a24a-09b4dd9f575b-host-cni-netd\") pod \"ovnkube-node-9dcqf\" (UID: \"b2481c54-52e5-44b3-a24a-09b4dd9f575b\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dcqf"
Sep 09 20:52:24.362855 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.360730    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/169cade8-7395-4b62-9544-ff0c47022a89-serviceca\") pod \"node-ca-tml69\" (UID: \"169cade8-7395-4b62-9544-ff0c47022a89\") " pod="openshift-image-registry/node-ca-tml69"
Sep 09 20:52:24.362855 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.360757    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b2481c54-52e5-44b3-a24a-09b4dd9f575b-var-lib-openvswitch\") pod \"ovnkube-node-9dcqf\" (UID: \"b2481c54-52e5-44b3-a24a-09b4dd9f575b\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dcqf"
Sep 09 20:52:24.362855 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.360759    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b2481c54-52e5-44b3-a24a-09b4dd9f575b-host-run-netns\") pod \"ovnkube-node-9dcqf\" (UID: \"b2481c54-52e5-44b3-a24a-09b4dd9f575b\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dcqf"
Sep 09 20:52:24.362855 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.360795    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b2481c54-52e5-44b3-a24a-09b4dd9f575b-host-run-netns\") pod \"ovnkube-node-9dcqf\" (UID: \"b2481c54-52e5-44b3-a24a-09b4dd9f575b\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dcqf"
Sep 09 20:52:24.362855 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.360871    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b2481c54-52e5-44b3-a24a-09b4dd9f575b-run-ovn\") pod \"ovnkube-node-9dcqf\" (UID: \"b2481c54-52e5-44b3-a24a-09b4dd9f575b\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dcqf"
Sep 09 20:52:24.362855 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.360949    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b2481c54-52e5-44b3-a24a-09b4dd9f575b-host-cni-netd\") pod \"ovnkube-node-9dcqf\" (UID: \"b2481c54-52e5-44b3-a24a-09b4dd9f575b\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dcqf"
Sep 09 20:52:24.363306 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.360987    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b2481c54-52e5-44b3-a24a-09b4dd9f575b-log-socket\") pod \"ovnkube-node-9dcqf\" (UID: \"b2481c54-52e5-44b3-a24a-09b4dd9f575b\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dcqf"
Sep 09 20:52:24.363306 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.360802    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b2481c54-52e5-44b3-a24a-09b4dd9f575b-ovnkube-config\") pod \"ovnkube-node-9dcqf\" (UID: \"b2481c54-52e5-44b3-a24a-09b4dd9f575b\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dcqf"
Sep 09 20:52:24.363306 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.361025    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/10f78cd8-5259-4024-ae5f-68bfa6a0626f-sys\") pod \"tuned-s98q4\" (UID: \"10f78cd8-5259-4024-ae5f-68bfa6a0626f\") " pod="openshift-cluster-node-tuning-operator/tuned-s98q4"
Sep 09 20:52:24.363306 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.361050    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/28da7add-257c-49f6-a846-6e8116b68d4c-host-slash\") pod \"iptables-alerter-kfpbw\" (UID: \"28da7add-257c-49f6-a846-6e8116b68d4c\") " pod="openshift-network-operator/iptables-alerter-kfpbw"
Sep 09 20:52:24.363306 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.361121    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/28da7add-257c-49f6-a846-6e8116b68d4c-host-slash\") pod \"iptables-alerter-kfpbw\" (UID: \"28da7add-257c-49f6-a846-6e8116b68d4c\") " pod="openshift-network-operator/iptables-alerter-kfpbw"
Sep 09 20:52:24.363306 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.361166    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/10f78cd8-5259-4024-ae5f-68bfa6a0626f-sys\") pod \"tuned-s98q4\" (UID: \"10f78cd8-5259-4024-ae5f-68bfa6a0626f\") " pod="openshift-cluster-node-tuning-operator/tuned-s98q4"
Sep 09 20:52:24.363306 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.360466    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a55d0249-76a1-42d0-8e06-8c4fa1e557f8-socket-dir\") pod \"aws-ebs-csi-driver-node-pdpk7\" (UID: \"a55d0249-76a1-42d0-8e06-8c4fa1e557f8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pdpk7"
Sep 09 20:52:24.363306 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.361490    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/169cade8-7395-4b62-9544-ff0c47022a89-serviceca\") pod \"node-ca-tml69\" (UID: \"169cade8-7395-4b62-9544-ff0c47022a89\") " pod="openshift-image-registry/node-ca-tml69"
Sep 09 20:52:24.363306 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.361500    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b2481c54-52e5-44b3-a24a-09b4dd9f575b-ovnkube-config\") pod \"ovnkube-node-9dcqf\" (UID: \"b2481c54-52e5-44b3-a24a-09b4dd9f575b\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dcqf"
Sep 09 20:52:24.363306 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.361704    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/10f78cd8-5259-4024-ae5f-68bfa6a0626f-etc-tuned\") pod \"tuned-s98q4\" (UID: \"10f78cd8-5259-4024-ae5f-68bfa6a0626f\") " pod="openshift-cluster-node-tuning-operator/tuned-s98q4"
Sep 09 20:52:24.363306 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.361906    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/28da7add-257c-49f6-a846-6e8116b68d4c-iptables-alerter-script\") pod \"iptables-alerter-kfpbw\" (UID: \"28da7add-257c-49f6-a846-6e8116b68d4c\") " pod="openshift-network-operator/iptables-alerter-kfpbw"
Sep 09 20:52:24.363306 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.362396    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/10f78cd8-5259-4024-ae5f-68bfa6a0626f-tmp\") pod \"tuned-s98q4\" (UID: \"10f78cd8-5259-4024-ae5f-68bfa6a0626f\") " pod="openshift-cluster-node-tuning-operator/tuned-s98q4"
Sep 09 20:52:24.363306 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.362769    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9075ffbd-9633-49ac-b6aa-f410776c4ad6-agent-certs\") pod \"konnectivity-agent-kwvw7\" (UID: \"9075ffbd-9633-49ac-b6aa-f410776c4ad6\") " pod="kube-system/konnectivity-agent-kwvw7"
Sep 09 20:52:24.363306 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.363031    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b2481c54-52e5-44b3-a24a-09b4dd9f575b-ovn-node-metrics-cert\") pod \"ovnkube-node-9dcqf\" (UID: \"b2481c54-52e5-44b3-a24a-09b4dd9f575b\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dcqf"
Sep 09 20:52:24.377071 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.377049    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tnkxj\" (UniqueName: \"kubernetes.io/projected/28da7add-257c-49f6-a846-6e8116b68d4c-kube-api-access-tnkxj\") pod \"iptables-alerter-kfpbw\" (UID: \"28da7add-257c-49f6-a846-6e8116b68d4c\") " pod="openshift-network-operator/iptables-alerter-kfpbw"
Sep 09 20:52:24.379974 ip-10-0-138-224 kubenswrapper[2579]: E0909 20:52:24.379954    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 20:52:24.380066 ip-10-0-138-224 kubenswrapper[2579]: E0909 20:52:24.379977    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 20:52:24.380066 ip-10-0-138-224 kubenswrapper[2579]: E0909 20:52:24.379991    2579 projected.go:196] Error preparing data for projected volume kube-api-access-6c58d for pod openshift-network-diagnostics/network-check-target-b6qr7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:52:24.380066 ip-10-0-138-224 kubenswrapper[2579]: E0909 20:52:24.380049    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e1193e63-dea9-41f4-a6ab-748d8ae109e8-kube-api-access-6c58d podName:e1193e63-dea9-41f4-a6ab-748d8ae109e8 nodeName:}" failed. No retries permitted until 2026-09-09 20:52:24.880032796 +0000 UTC m=+2.120126912 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-6c58d" (UniqueName: "kubernetes.io/projected/e1193e63-dea9-41f4-a6ab-748d8ae109e8-kube-api-access-6c58d") pod "network-check-target-b6qr7" (UID: "e1193e63-dea9-41f4-a6ab-748d8ae109e8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:52:24.381834 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.381813    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdgb2\" (UniqueName: \"kubernetes.io/projected/b2481c54-52e5-44b3-a24a-09b4dd9f575b-kube-api-access-bdgb2\") pod \"ovnkube-node-9dcqf\" (UID: \"b2481c54-52e5-44b3-a24a-09b4dd9f575b\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dcqf"
Sep 09 20:52:24.382088 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.382057    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5r9bj\" (UniqueName: \"kubernetes.io/projected/169cade8-7395-4b62-9544-ff0c47022a89-kube-api-access-5r9bj\") pod \"node-ca-tml69\" (UID: \"169cade8-7395-4b62-9544-ff0c47022a89\") " pod="openshift-image-registry/node-ca-tml69"
Sep 09 20:52:24.382088 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.382071    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-crvp7\" (UniqueName: \"kubernetes.io/projected/c6fd107c-ea2f-4e85-9a32-1e11f0858fd3-kube-api-access-crvp7\") pod \"network-metrics-daemon-ws8hc\" (UID: \"c6fd107c-ea2f-4e85-9a32-1e11f0858fd3\") " pod="openshift-multus/network-metrics-daemon-ws8hc"
Sep 09 20:52:24.382329 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.382312    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmgq4\" (UniqueName: \"kubernetes.io/projected/10f78cd8-5259-4024-ae5f-68bfa6a0626f-kube-api-access-gmgq4\") pod \"tuned-s98q4\" (UID: \"10f78cd8-5259-4024-ae5f-68bfa6a0626f\") " pod="openshift-cluster-node-tuning-operator/tuned-s98q4"
Sep 09 20:52:24.383844 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.383824    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwfng\" (UniqueName: \"kubernetes.io/projected/a55d0249-76a1-42d0-8e06-8c4fa1e557f8-kube-api-access-pwfng\") pod \"aws-ebs-csi-driver-node-pdpk7\" (UID: \"a55d0249-76a1-42d0-8e06-8c4fa1e557f8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pdpk7"
Sep 09 20:52:24.477224 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.477165    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-88pfs"
Sep 09 20:52:24.493450 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.493426    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-9rprp"
Sep 09 20:52:24.513044 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.513024    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-jbs9t"
Sep 09 20:52:24.525659 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.525640    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-9dcqf"
Sep 09 20:52:24.542219 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.542201    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pdpk7"
Sep 09 20:52:24.557973 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.557951    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-s98q4"
Sep 09 20:52:24.574956 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.574942    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-tml69"
Sep 09 20:52:24.596489 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.596468    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-kfpbw"
Sep 09 20:52:24.600872 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.600857    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-kwvw7"
Sep 09 20:52:24.641715 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.641696    2579 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 20:52:24.812082 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:24.812038    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda125a7c81ad75870351cc3dc16d80838.slice/crio-660c8eca537f28b41901a35e44aa3d551cb5beaaf0f7c26837b2e8fa644f09df WatchSource:0}: Error finding container 660c8eca537f28b41901a35e44aa3d551cb5beaaf0f7c26837b2e8fa644f09df: Status 404 returned error can't find the container with id 660c8eca537f28b41901a35e44aa3d551cb5beaaf0f7c26837b2e8fa644f09df
Sep 09 20:52:24.815995 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.815980    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 20:52:24.854267 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:24.854244    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfe1a979_9dff_419b_b517_93e03c19c040.slice/crio-1423bcfba18f1305b94d03a68402d0ea460aca2aee0a81edeeb53d5476abb0e1 WatchSource:0}: Error finding container 1423bcfba18f1305b94d03a68402d0ea460aca2aee0a81edeeb53d5476abb0e1: Status 404 returned error can't find the container with id 1423bcfba18f1305b94d03a68402d0ea460aca2aee0a81edeeb53d5476abb0e1
Sep 09 20:52:24.864031 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.864012    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/732d9d71-d680-4291-aaee-028307ef79dc-original-pull-secret\") pod \"global-pull-secret-syncer-gz9zx\" (UID: \"732d9d71-d680-4291-aaee-028307ef79dc\") " pod="kube-system/global-pull-secret-syncer-gz9zx"
Sep 09 20:52:24.864123 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.864072    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c6fd107c-ea2f-4e85-9a32-1e11f0858fd3-metrics-certs\") pod \"network-metrics-daemon-ws8hc\" (UID: \"c6fd107c-ea2f-4e85-9a32-1e11f0858fd3\") " pod="openshift-multus/network-metrics-daemon-ws8hc"
Sep 09 20:52:24.864183 ip-10-0-138-224 kubenswrapper[2579]: E0909 20:52:24.864159    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 20:52:24.864183 ip-10-0-138-224 kubenswrapper[2579]: E0909 20:52:24.864164    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:52:24.864251 ip-10-0-138-224 kubenswrapper[2579]: E0909 20:52:24.864221    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c6fd107c-ea2f-4e85-9a32-1e11f0858fd3-metrics-certs podName:c6fd107c-ea2f-4e85-9a32-1e11f0858fd3 nodeName:}" failed. No retries permitted until 2026-09-09 20:52:25.864204286 +0000 UTC m=+3.104298394 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c6fd107c-ea2f-4e85-9a32-1e11f0858fd3-metrics-certs") pod "network-metrics-daemon-ws8hc" (UID: "c6fd107c-ea2f-4e85-9a32-1e11f0858fd3") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:52:24.864251 ip-10-0-138-224 kubenswrapper[2579]: E0909 20:52:24.864240    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/732d9d71-d680-4291-aaee-028307ef79dc-original-pull-secret podName:732d9d71-d680-4291-aaee-028307ef79dc nodeName:}" failed. No retries permitted until 2026-09-09 20:52:25.864229717 +0000 UTC m=+3.104323813 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/732d9d71-d680-4291-aaee-028307ef79dc-original-pull-secret") pod "global-pull-secret-syncer-gz9zx" (UID: "732d9d71-d680-4291-aaee-028307ef79dc") : object "kube-system"/"original-pull-secret" not registered
Sep 09 20:52:24.916575 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:24.916420    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10f78cd8_5259_4024_ae5f_68bfa6a0626f.slice/crio-8ce7ff1625e49078160ffd77db16b6cfa09f2f6cbffc0d66a35c6c2baa8d7e29 WatchSource:0}: Error finding container 8ce7ff1625e49078160ffd77db16b6cfa09f2f6cbffc0d66a35c6c2baa8d7e29: Status 404 returned error can't find the container with id 8ce7ff1625e49078160ffd77db16b6cfa09f2f6cbffc0d66a35c6c2baa8d7e29
Sep 09 20:52:24.927243 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:24.927226    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2481c54_52e5_44b3_a24a_09b4dd9f575b.slice/crio-a2d97a83ab484f2dda22aed40319b7bf37f2691a8d959494ac59f66541ce41bc WatchSource:0}: Error finding container a2d97a83ab484f2dda22aed40319b7bf37f2691a8d959494ac59f66541ce41bc: Status 404 returned error can't find the container with id a2d97a83ab484f2dda22aed40319b7bf37f2691a8d959494ac59f66541ce41bc
Sep 09 20:52:24.949576 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:24.949555    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod169cade8_7395_4b62_9544_ff0c47022a89.slice/crio-57c85b890d0401d23a6ea0524301e6469cc170386b013b0ae170f25b59bd3bb6 WatchSource:0}: Error finding container 57c85b890d0401d23a6ea0524301e6469cc170386b013b0ae170f25b59bd3bb6: Status 404 returned error can't find the container with id 57c85b890d0401d23a6ea0524301e6469cc170386b013b0ae170f25b59bd3bb6
Sep 09 20:52:24.964341 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.964317    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6c58d\" (UniqueName: \"kubernetes.io/projected/e1193e63-dea9-41f4-a6ab-748d8ae109e8-kube-api-access-6c58d\") pod \"network-check-target-b6qr7\" (UID: \"e1193e63-dea9-41f4-a6ab-748d8ae109e8\") " pod="openshift-network-diagnostics/network-check-target-b6qr7"
Sep 09 20:52:24.964643 ip-10-0-138-224 kubenswrapper[2579]: E0909 20:52:24.964623    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 20:52:24.964719 ip-10-0-138-224 kubenswrapper[2579]: E0909 20:52:24.964649    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 20:52:24.964719 ip-10-0-138-224 kubenswrapper[2579]: E0909 20:52:24.964660    2579 projected.go:196] Error preparing data for projected volume kube-api-access-6c58d for pod openshift-network-diagnostics/network-check-target-b6qr7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:52:24.964719 ip-10-0-138-224 kubenswrapper[2579]: E0909 20:52:24.964709    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e1193e63-dea9-41f4-a6ab-748d8ae109e8-kube-api-access-6c58d podName:e1193e63-dea9-41f4-a6ab-748d8ae109e8 nodeName:}" failed. No retries permitted until 2026-09-09 20:52:25.964690827 +0000 UTC m=+3.204784937 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-6c58d" (UniqueName: "kubernetes.io/projected/e1193e63-dea9-41f4-a6ab-748d8ae109e8-kube-api-access-6c58d") pod "network-check-target-b6qr7" (UID: "e1193e63-dea9-41f4-a6ab-748d8ae109e8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:52:24.966913 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:24.966878    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda55d0249_76a1_42d0_8e06_8c4fa1e557f8.slice/crio-9e43c72084ba08d5c2534dc20edc7eb7fe1b8cf1de1d99d78412cb619816f8af WatchSource:0}: Error finding container 9e43c72084ba08d5c2534dc20edc7eb7fe1b8cf1de1d99d78412cb619816f8af: Status 404 returned error can't find the container with id 9e43c72084ba08d5c2534dc20edc7eb7fe1b8cf1de1d99d78412cb619816f8af
Sep 09 20:52:24.993618 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:24.993602    2579 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 20:52:25.012931 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:25.012909    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e4d14d2e217a05b09faab3ef55f4d0f.slice/crio-604b96386a3e244b094c575a5aa65ec5aabb6e8cbd8f9ddf1b8e1709f0ed0be1 WatchSource:0}: Error finding container 604b96386a3e244b094c575a5aa65ec5aabb6e8cbd8f9ddf1b8e1709f0ed0be1: Status 404 returned error can't find the container with id 604b96386a3e244b094c575a5aa65ec5aabb6e8cbd8f9ddf1b8e1709f0ed0be1
Sep 09 20:52:25.015673 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:25.015648    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9075ffbd_9633_49ac_b6aa_f410776c4ad6.slice/crio-d291e47a648a3f1e46dde9d42854d4204f49a66636122f90b29d46f4a6a63720 WatchSource:0}: Error finding container d291e47a648a3f1e46dde9d42854d4204f49a66636122f90b29d46f4a6a63720: Status 404 returned error can't find the container with id d291e47a648a3f1e46dde9d42854d4204f49a66636122f90b29d46f4a6a63720
Sep 09 20:52:25.016158 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:25.016138    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22f66b47_36ad_4226_b0f9_ad4075eaaa2d.slice/crio-72a229c46c4212d5d040ca659113cdd0b35c804cf7c1baa5a19ac36314f8bc85 WatchSource:0}: Error finding container 72a229c46c4212d5d040ca659113cdd0b35c804cf7c1baa5a19ac36314f8bc85: Status 404 returned error can't find the container with id 72a229c46c4212d5d040ca659113cdd0b35c804cf7c1baa5a19ac36314f8bc85
Sep 09 20:52:25.062010 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:25.061992    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddb26243_140d_4ac9_8a3d_4079559bd0e7.slice/crio-da75f8f648a880606599d3dfdd4714928971af4a7c06574ea58e3f0cd4ce59dc WatchSource:0}: Error finding container da75f8f648a880606599d3dfdd4714928971af4a7c06574ea58e3f0cd4ce59dc: Status 404 returned error can't find the container with id da75f8f648a880606599d3dfdd4714928971af4a7c06574ea58e3f0cd4ce59dc
Sep 09 20:52:25.198323 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:25.198278    2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-08 20:47:24 +0000 UTC" deadline="2028-04-21 17:12:46.251956333 +0000 UTC"
Sep 09 20:52:25.198323 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:25.198322    2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14156h20m21.05363789s"
Sep 09 20:52:25.285428 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:25.285232    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ws8hc"
Sep 09 20:52:25.285428 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:25.285248    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b6qr7"
Sep 09 20:52:25.285428 ip-10-0-138-224 kubenswrapper[2579]: E0909 20:52:25.285369    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ws8hc" podUID="c6fd107c-ea2f-4e85-9a32-1e11f0858fd3"
Sep 09 20:52:25.285428 ip-10-0-138-224 kubenswrapper[2579]: E0909 20:52:25.285393    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b6qr7" podUID="e1193e63-dea9-41f4-a6ab-748d8ae109e8"
Sep 09 20:52:25.289855 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:25.289782    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-jbs9t" event={"ID":"22f66b47-36ad-4226-b0f9-ad4075eaaa2d","Type":"ContainerStarted","Data":"72a229c46c4212d5d040ca659113cdd0b35c804cf7c1baa5a19ac36314f8bc85"}
Sep 09 20:52:25.291619 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:25.291596    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-kwvw7" event={"ID":"9075ffbd-9633-49ac-b6aa-f410776c4ad6","Type":"ContainerStarted","Data":"d291e47a648a3f1e46dde9d42854d4204f49a66636122f90b29d46f4a6a63720"}
Sep 09 20:52:25.293056 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:25.293033    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pdpk7" event={"ID":"a55d0249-76a1-42d0-8e06-8c4fa1e557f8","Type":"ContainerStarted","Data":"9e43c72084ba08d5c2534dc20edc7eb7fe1b8cf1de1d99d78412cb619816f8af"}
Sep 09 20:52:25.294624 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:25.294562    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-tml69" event={"ID":"169cade8-7395-4b62-9544-ff0c47022a89","Type":"ContainerStarted","Data":"57c85b890d0401d23a6ea0524301e6469cc170386b013b0ae170f25b59bd3bb6"}
Sep 09 20:52:25.295940 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:25.295918    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9dcqf" event={"ID":"b2481c54-52e5-44b3-a24a-09b4dd9f575b","Type":"ContainerStarted","Data":"a2d97a83ab484f2dda22aed40319b7bf37f2691a8d959494ac59f66541ce41bc"}
Sep 09 20:52:25.297268 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:25.297249    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-s98q4" event={"ID":"10f78cd8-5259-4024-ae5f-68bfa6a0626f","Type":"ContainerStarted","Data":"8ce7ff1625e49078160ffd77db16b6cfa09f2f6cbffc0d66a35c6c2baa8d7e29"}
Sep 09 20:52:25.298551 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:25.298526    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9rprp" event={"ID":"dfe1a979-9dff-419b-b517-93e03c19c040","Type":"ContainerStarted","Data":"1423bcfba18f1305b94d03a68402d0ea460aca2aee0a81edeeb53d5476abb0e1"}
Sep 09 20:52:25.299978 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:25.299958    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-224.ec2.internal" event={"ID":"a125a7c81ad75870351cc3dc16d80838","Type":"ContainerStarted","Data":"660c8eca537f28b41901a35e44aa3d551cb5beaaf0f7c26837b2e8fa644f09df"}
Sep 09 20:52:25.301372 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:25.301351    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-88pfs" event={"ID":"ddb26243-140d-4ac9-8a3d-4079559bd0e7","Type":"ContainerStarted","Data":"da75f8f648a880606599d3dfdd4714928971af4a7c06574ea58e3f0cd4ce59dc"}
Sep 09 20:52:25.302736 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:25.302713    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-224.ec2.internal" event={"ID":"7e4d14d2e217a05b09faab3ef55f4d0f","Type":"ContainerStarted","Data":"604b96386a3e244b094c575a5aa65ec5aabb6e8cbd8f9ddf1b8e1709f0ed0be1"}
Sep 09 20:52:25.871753 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:25.871716    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c6fd107c-ea2f-4e85-9a32-1e11f0858fd3-metrics-certs\") pod \"network-metrics-daemon-ws8hc\" (UID: \"c6fd107c-ea2f-4e85-9a32-1e11f0858fd3\") " pod="openshift-multus/network-metrics-daemon-ws8hc"
Sep 09 20:52:25.871923 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:25.871788    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/732d9d71-d680-4291-aaee-028307ef79dc-original-pull-secret\") pod \"global-pull-secret-syncer-gz9zx\" (UID: \"732d9d71-d680-4291-aaee-028307ef79dc\") " pod="kube-system/global-pull-secret-syncer-gz9zx"
Sep 09 20:52:25.871923 ip-10-0-138-224 kubenswrapper[2579]: E0909 20:52:25.871871    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:52:25.871923 ip-10-0-138-224 kubenswrapper[2579]: E0909 20:52:25.871910    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 20:52:25.872115 ip-10-0-138-224 kubenswrapper[2579]: E0909 20:52:25.871947    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c6fd107c-ea2f-4e85-9a32-1e11f0858fd3-metrics-certs podName:c6fd107c-ea2f-4e85-9a32-1e11f0858fd3 nodeName:}" failed. No retries permitted until 2026-09-09 20:52:27.871925302 +0000 UTC m=+5.112019415 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c6fd107c-ea2f-4e85-9a32-1e11f0858fd3-metrics-certs") pod "network-metrics-daemon-ws8hc" (UID: "c6fd107c-ea2f-4e85-9a32-1e11f0858fd3") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:52:25.872115 ip-10-0-138-224 kubenswrapper[2579]: E0909 20:52:25.871980    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/732d9d71-d680-4291-aaee-028307ef79dc-original-pull-secret podName:732d9d71-d680-4291-aaee-028307ef79dc nodeName:}" failed. No retries permitted until 2026-09-09 20:52:27.871954803 +0000 UTC m=+5.112048912 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/732d9d71-d680-4291-aaee-028307ef79dc-original-pull-secret") pod "global-pull-secret-syncer-gz9zx" (UID: "732d9d71-d680-4291-aaee-028307ef79dc") : object "kube-system"/"original-pull-secret" not registered
Sep 09 20:52:25.972361 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:25.972323    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6c58d\" (UniqueName: \"kubernetes.io/projected/e1193e63-dea9-41f4-a6ab-748d8ae109e8-kube-api-access-6c58d\") pod \"network-check-target-b6qr7\" (UID: \"e1193e63-dea9-41f4-a6ab-748d8ae109e8\") " pod="openshift-network-diagnostics/network-check-target-b6qr7"
Sep 09 20:52:25.972579 ip-10-0-138-224 kubenswrapper[2579]: E0909 20:52:25.972534    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 20:52:25.972579 ip-10-0-138-224 kubenswrapper[2579]: E0909 20:52:25.972558    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 20:52:25.972579 ip-10-0-138-224 kubenswrapper[2579]: E0909 20:52:25.972572    2579 projected.go:196] Error preparing data for projected volume kube-api-access-6c58d for pod openshift-network-diagnostics/network-check-target-b6qr7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:52:25.972854 ip-10-0-138-224 kubenswrapper[2579]: E0909 20:52:25.972634    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e1193e63-dea9-41f4-a6ab-748d8ae109e8-kube-api-access-6c58d podName:e1193e63-dea9-41f4-a6ab-748d8ae109e8 nodeName:}" failed. No retries permitted until 2026-09-09 20:52:27.972616307 +0000 UTC m=+5.212710407 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-6c58d" (UniqueName: "kubernetes.io/projected/e1193e63-dea9-41f4-a6ab-748d8ae109e8-kube-api-access-6c58d") pod "network-check-target-b6qr7" (UID: "e1193e63-dea9-41f4-a6ab-748d8ae109e8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:52:26.199167 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:26.199072    2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-08 20:47:24 +0000 UTC" deadline="2028-02-08 13:58:45.22958343 +0000 UTC"
Sep 09 20:52:26.199167 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:26.199115    2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12401h6m19.030473376s"
Sep 09 20:52:26.228355 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:26.228325    2579 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 20:52:26.286585 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:26.286012    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gz9zx"
Sep 09 20:52:26.286585 ip-10-0-138-224 kubenswrapper[2579]: E0909 20:52:26.286159    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gz9zx" podUID="732d9d71-d680-4291-aaee-028307ef79dc"
Sep 09 20:52:26.337839 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:26.337563    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-kfpbw" event={"ID":"28da7add-257c-49f6-a846-6e8116b68d4c","Type":"ContainerStarted","Data":"825bfc584e6518bfbf7e76fc044729d86e080dde635ed256e8ea278118c1e6ad"}
Sep 09 20:52:27.290450 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:27.290364    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b6qr7"
Sep 09 20:52:27.290979 ip-10-0-138-224 kubenswrapper[2579]: E0909 20:52:27.290507    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b6qr7" podUID="e1193e63-dea9-41f4-a6ab-748d8ae109e8"
Sep 09 20:52:27.290979 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:27.290862    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ws8hc"
Sep 09 20:52:27.290979 ip-10-0-138-224 kubenswrapper[2579]: E0909 20:52:27.290972    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ws8hc" podUID="c6fd107c-ea2f-4e85-9a32-1e11f0858fd3"
Sep 09 20:52:27.888379 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:27.888340    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c6fd107c-ea2f-4e85-9a32-1e11f0858fd3-metrics-certs\") pod \"network-metrics-daemon-ws8hc\" (UID: \"c6fd107c-ea2f-4e85-9a32-1e11f0858fd3\") " pod="openshift-multus/network-metrics-daemon-ws8hc"
Sep 09 20:52:27.888590 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:27.888420    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/732d9d71-d680-4291-aaee-028307ef79dc-original-pull-secret\") pod \"global-pull-secret-syncer-gz9zx\" (UID: \"732d9d71-d680-4291-aaee-028307ef79dc\") " pod="kube-system/global-pull-secret-syncer-gz9zx"
Sep 09 20:52:27.888590 ip-10-0-138-224 kubenswrapper[2579]: E0909 20:52:27.888578    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 20:52:27.888693 ip-10-0-138-224 kubenswrapper[2579]: E0909 20:52:27.888647    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/732d9d71-d680-4291-aaee-028307ef79dc-original-pull-secret podName:732d9d71-d680-4291-aaee-028307ef79dc nodeName:}" failed. No retries permitted until 2026-09-09 20:52:31.888627201 +0000 UTC m=+9.128721326 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/732d9d71-d680-4291-aaee-028307ef79dc-original-pull-secret") pod "global-pull-secret-syncer-gz9zx" (UID: "732d9d71-d680-4291-aaee-028307ef79dc") : object "kube-system"/"original-pull-secret" not registered
Sep 09 20:52:27.889184 ip-10-0-138-224 kubenswrapper[2579]: E0909 20:52:27.889160    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:52:27.889282 ip-10-0-138-224 kubenswrapper[2579]: E0909 20:52:27.889226    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c6fd107c-ea2f-4e85-9a32-1e11f0858fd3-metrics-certs podName:c6fd107c-ea2f-4e85-9a32-1e11f0858fd3 nodeName:}" failed. No retries permitted until 2026-09-09 20:52:31.889208048 +0000 UTC m=+9.129302163 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c6fd107c-ea2f-4e85-9a32-1e11f0858fd3-metrics-certs") pod "network-metrics-daemon-ws8hc" (UID: "c6fd107c-ea2f-4e85-9a32-1e11f0858fd3") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:52:27.989350 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:27.989311    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6c58d\" (UniqueName: \"kubernetes.io/projected/e1193e63-dea9-41f4-a6ab-748d8ae109e8-kube-api-access-6c58d\") pod \"network-check-target-b6qr7\" (UID: \"e1193e63-dea9-41f4-a6ab-748d8ae109e8\") " pod="openshift-network-diagnostics/network-check-target-b6qr7"
Sep 09 20:52:27.989547 ip-10-0-138-224 kubenswrapper[2579]: E0909 20:52:27.989505    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 20:52:27.989547 ip-10-0-138-224 kubenswrapper[2579]: E0909 20:52:27.989527    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 20:52:27.989547 ip-10-0-138-224 kubenswrapper[2579]: E0909 20:52:27.989541    2579 projected.go:196] Error preparing data for projected volume kube-api-access-6c58d for pod openshift-network-diagnostics/network-check-target-b6qr7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:52:27.989705 ip-10-0-138-224 kubenswrapper[2579]: E0909 20:52:27.989600    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e1193e63-dea9-41f4-a6ab-748d8ae109e8-kube-api-access-6c58d podName:e1193e63-dea9-41f4-a6ab-748d8ae109e8 nodeName:}" failed. No retries permitted until 2026-09-09 20:52:31.989581439 +0000 UTC m=+9.229675546 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-6c58d" (UniqueName: "kubernetes.io/projected/e1193e63-dea9-41f4-a6ab-748d8ae109e8-kube-api-access-6c58d") pod "network-check-target-b6qr7" (UID: "e1193e63-dea9-41f4-a6ab-748d8ae109e8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:52:28.285846 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:28.285755    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gz9zx"
Sep 09 20:52:28.286006 ip-10-0-138-224 kubenswrapper[2579]: E0909 20:52:28.285893    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gz9zx" podUID="732d9d71-d680-4291-aaee-028307ef79dc"
Sep 09 20:52:29.285596 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:29.285563    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b6qr7"
Sep 09 20:52:29.286044 ip-10-0-138-224 kubenswrapper[2579]: E0909 20:52:29.285703    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b6qr7" podUID="e1193e63-dea9-41f4-a6ab-748d8ae109e8"
Sep 09 20:52:29.286044 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:29.285748    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ws8hc"
Sep 09 20:52:29.286044 ip-10-0-138-224 kubenswrapper[2579]: E0909 20:52:29.285840    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ws8hc" podUID="c6fd107c-ea2f-4e85-9a32-1e11f0858fd3"
Sep 09 20:52:30.285284 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:30.285244    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gz9zx"
Sep 09 20:52:30.285482 ip-10-0-138-224 kubenswrapper[2579]: E0909 20:52:30.285378    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gz9zx" podUID="732d9d71-d680-4291-aaee-028307ef79dc"
Sep 09 20:52:31.285836 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:31.285801    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b6qr7"
Sep 09 20:52:31.286224 ip-10-0-138-224 kubenswrapper[2579]: E0909 20:52:31.285941    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b6qr7" podUID="e1193e63-dea9-41f4-a6ab-748d8ae109e8"
Sep 09 20:52:31.286224 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:31.285988    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ws8hc"
Sep 09 20:52:31.286224 ip-10-0-138-224 kubenswrapper[2579]: E0909 20:52:31.286143    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ws8hc" podUID="c6fd107c-ea2f-4e85-9a32-1e11f0858fd3"
Sep 09 20:52:31.924420 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:31.923995    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/732d9d71-d680-4291-aaee-028307ef79dc-original-pull-secret\") pod \"global-pull-secret-syncer-gz9zx\" (UID: \"732d9d71-d680-4291-aaee-028307ef79dc\") " pod="kube-system/global-pull-secret-syncer-gz9zx"
Sep 09 20:52:31.924420 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:31.924094    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c6fd107c-ea2f-4e85-9a32-1e11f0858fd3-metrics-certs\") pod \"network-metrics-daemon-ws8hc\" (UID: \"c6fd107c-ea2f-4e85-9a32-1e11f0858fd3\") " pod="openshift-multus/network-metrics-daemon-ws8hc"
Sep 09 20:52:31.924420 ip-10-0-138-224 kubenswrapper[2579]: E0909 20:52:31.924239    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:52:31.924420 ip-10-0-138-224 kubenswrapper[2579]: E0909 20:52:31.924304    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c6fd107c-ea2f-4e85-9a32-1e11f0858fd3-metrics-certs podName:c6fd107c-ea2f-4e85-9a32-1e11f0858fd3 nodeName:}" failed. No retries permitted until 2026-09-09 20:52:39.924283252 +0000 UTC m=+17.164377365 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c6fd107c-ea2f-4e85-9a32-1e11f0858fd3-metrics-certs") pod "network-metrics-daemon-ws8hc" (UID: "c6fd107c-ea2f-4e85-9a32-1e11f0858fd3") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:52:31.924420 ip-10-0-138-224 kubenswrapper[2579]: E0909 20:52:31.924322    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 20:52:31.924420 ip-10-0-138-224 kubenswrapper[2579]: E0909 20:52:31.924361    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/732d9d71-d680-4291-aaee-028307ef79dc-original-pull-secret podName:732d9d71-d680-4291-aaee-028307ef79dc nodeName:}" failed. No retries permitted until 2026-09-09 20:52:39.924349639 +0000 UTC m=+17.164443750 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/732d9d71-d680-4291-aaee-028307ef79dc-original-pull-secret") pod "global-pull-secret-syncer-gz9zx" (UID: "732d9d71-d680-4291-aaee-028307ef79dc") : object "kube-system"/"original-pull-secret" not registered
Sep 09 20:52:32.025351 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:32.025295    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6c58d\" (UniqueName: \"kubernetes.io/projected/e1193e63-dea9-41f4-a6ab-748d8ae109e8-kube-api-access-6c58d\") pod \"network-check-target-b6qr7\" (UID: \"e1193e63-dea9-41f4-a6ab-748d8ae109e8\") " pod="openshift-network-diagnostics/network-check-target-b6qr7"
Sep 09 20:52:32.025562 ip-10-0-138-224 kubenswrapper[2579]: E0909 20:52:32.025536    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 20:52:32.025562 ip-10-0-138-224 kubenswrapper[2579]: E0909 20:52:32.025563    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 20:52:32.025730 ip-10-0-138-224 kubenswrapper[2579]: E0909 20:52:32.025576    2579 projected.go:196] Error preparing data for projected volume kube-api-access-6c58d for pod openshift-network-diagnostics/network-check-target-b6qr7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:52:32.025730 ip-10-0-138-224 kubenswrapper[2579]: E0909 20:52:32.025638    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e1193e63-dea9-41f4-a6ab-748d8ae109e8-kube-api-access-6c58d podName:e1193e63-dea9-41f4-a6ab-748d8ae109e8 nodeName:}" failed. No retries permitted until 2026-09-09 20:52:40.025620166 +0000 UTC m=+17.265714266 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-6c58d" (UniqueName: "kubernetes.io/projected/e1193e63-dea9-41f4-a6ab-748d8ae109e8-kube-api-access-6c58d") pod "network-check-target-b6qr7" (UID: "e1193e63-dea9-41f4-a6ab-748d8ae109e8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:52:32.285783 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:32.285751    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gz9zx"
Sep 09 20:52:32.285974 ip-10-0-138-224 kubenswrapper[2579]: E0909 20:52:32.285876    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gz9zx" podUID="732d9d71-d680-4291-aaee-028307ef79dc"
Sep 09 20:52:33.289545 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:33.289506    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b6qr7"
Sep 09 20:52:33.289545 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:33.289541    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ws8hc"
Sep 09 20:52:33.290156 ip-10-0-138-224 kubenswrapper[2579]: E0909 20:52:33.289632    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b6qr7" podUID="e1193e63-dea9-41f4-a6ab-748d8ae109e8"
Sep 09 20:52:33.290156 ip-10-0-138-224 kubenswrapper[2579]: E0909 20:52:33.289731    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ws8hc" podUID="c6fd107c-ea2f-4e85-9a32-1e11f0858fd3"
Sep 09 20:52:34.285676 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:34.285639    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gz9zx"
Sep 09 20:52:34.285822 ip-10-0-138-224 kubenswrapper[2579]: E0909 20:52:34.285750    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gz9zx" podUID="732d9d71-d680-4291-aaee-028307ef79dc"
Sep 09 20:52:35.288889 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:35.288851    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ws8hc"
Sep 09 20:52:35.288889 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:35.288889    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b6qr7"
Sep 09 20:52:35.289385 ip-10-0-138-224 kubenswrapper[2579]: E0909 20:52:35.288993    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ws8hc" podUID="c6fd107c-ea2f-4e85-9a32-1e11f0858fd3"
Sep 09 20:52:35.289385 ip-10-0-138-224 kubenswrapper[2579]: E0909 20:52:35.289045    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b6qr7" podUID="e1193e63-dea9-41f4-a6ab-748d8ae109e8"
Sep 09 20:52:36.285508 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:36.285471    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gz9zx"
Sep 09 20:52:36.285685 ip-10-0-138-224 kubenswrapper[2579]: E0909 20:52:36.285590    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gz9zx" podUID="732d9d71-d680-4291-aaee-028307ef79dc"
Sep 09 20:52:37.289135 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:37.288586    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b6qr7"
Sep 09 20:52:37.289135 ip-10-0-138-224 kubenswrapper[2579]: E0909 20:52:37.288719    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b6qr7" podUID="e1193e63-dea9-41f4-a6ab-748d8ae109e8"
Sep 09 20:52:37.289135 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:37.288769    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ws8hc"
Sep 09 20:52:37.289135 ip-10-0-138-224 kubenswrapper[2579]: E0909 20:52:37.288887    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ws8hc" podUID="c6fd107c-ea2f-4e85-9a32-1e11f0858fd3"
Sep 09 20:52:38.286016 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:38.285977    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gz9zx"
Sep 09 20:52:38.286190 ip-10-0-138-224 kubenswrapper[2579]: E0909 20:52:38.286098    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gz9zx" podUID="732d9d71-d680-4291-aaee-028307ef79dc"
Sep 09 20:52:39.285417 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:39.285373    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ws8hc"
Sep 09 20:52:39.285908 ip-10-0-138-224 kubenswrapper[2579]: E0909 20:52:39.285534    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ws8hc" podUID="c6fd107c-ea2f-4e85-9a32-1e11f0858fd3"
Sep 09 20:52:39.285908 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:39.285372    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b6qr7"
Sep 09 20:52:39.286046 ip-10-0-138-224 kubenswrapper[2579]: E0909 20:52:39.285940    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b6qr7" podUID="e1193e63-dea9-41f4-a6ab-748d8ae109e8"
Sep 09 20:52:39.985502 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:39.985456    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/732d9d71-d680-4291-aaee-028307ef79dc-original-pull-secret\") pod \"global-pull-secret-syncer-gz9zx\" (UID: \"732d9d71-d680-4291-aaee-028307ef79dc\") " pod="kube-system/global-pull-secret-syncer-gz9zx"
Sep 09 20:52:39.985679 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:39.985550    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c6fd107c-ea2f-4e85-9a32-1e11f0858fd3-metrics-certs\") pod \"network-metrics-daemon-ws8hc\" (UID: \"c6fd107c-ea2f-4e85-9a32-1e11f0858fd3\") " pod="openshift-multus/network-metrics-daemon-ws8hc"
Sep 09 20:52:39.985679 ip-10-0-138-224 kubenswrapper[2579]: E0909 20:52:39.985635    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 20:52:39.985679 ip-10-0-138-224 kubenswrapper[2579]: E0909 20:52:39.985674    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:52:39.985835 ip-10-0-138-224 kubenswrapper[2579]: E0909 20:52:39.985707    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/732d9d71-d680-4291-aaee-028307ef79dc-original-pull-secret podName:732d9d71-d680-4291-aaee-028307ef79dc nodeName:}" failed. No retries permitted until 2026-09-09 20:52:55.985691543 +0000 UTC m=+33.225785639 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/732d9d71-d680-4291-aaee-028307ef79dc-original-pull-secret") pod "global-pull-secret-syncer-gz9zx" (UID: "732d9d71-d680-4291-aaee-028307ef79dc") : object "kube-system"/"original-pull-secret" not registered
Sep 09 20:52:39.985835 ip-10-0-138-224 kubenswrapper[2579]: E0909 20:52:39.985727    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c6fd107c-ea2f-4e85-9a32-1e11f0858fd3-metrics-certs podName:c6fd107c-ea2f-4e85-9a32-1e11f0858fd3 nodeName:}" failed. No retries permitted until 2026-09-09 20:52:55.98571408 +0000 UTC m=+33.225808176 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c6fd107c-ea2f-4e85-9a32-1e11f0858fd3-metrics-certs") pod "network-metrics-daemon-ws8hc" (UID: "c6fd107c-ea2f-4e85-9a32-1e11f0858fd3") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:52:40.086285 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:40.086250    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6c58d\" (UniqueName: \"kubernetes.io/projected/e1193e63-dea9-41f4-a6ab-748d8ae109e8-kube-api-access-6c58d\") pod \"network-check-target-b6qr7\" (UID: \"e1193e63-dea9-41f4-a6ab-748d8ae109e8\") " pod="openshift-network-diagnostics/network-check-target-b6qr7"
Sep 09 20:52:40.086504 ip-10-0-138-224 kubenswrapper[2579]: E0909 20:52:40.086435    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 20:52:40.086504 ip-10-0-138-224 kubenswrapper[2579]: E0909 20:52:40.086481    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 20:52:40.086504 ip-10-0-138-224 kubenswrapper[2579]: E0909 20:52:40.086492    2579 projected.go:196] Error preparing data for projected volume kube-api-access-6c58d for pod openshift-network-diagnostics/network-check-target-b6qr7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:52:40.086641 ip-10-0-138-224 kubenswrapper[2579]: E0909 20:52:40.086550    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e1193e63-dea9-41f4-a6ab-748d8ae109e8-kube-api-access-6c58d podName:e1193e63-dea9-41f4-a6ab-748d8ae109e8 nodeName:}" failed. No retries permitted until 2026-09-09 20:52:56.086534243 +0000 UTC m=+33.326628339 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-6c58d" (UniqueName: "kubernetes.io/projected/e1193e63-dea9-41f4-a6ab-748d8ae109e8-kube-api-access-6c58d") pod "network-check-target-b6qr7" (UID: "e1193e63-dea9-41f4-a6ab-748d8ae109e8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:52:40.285651 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:40.285549    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gz9zx"
Sep 09 20:52:40.286131 ip-10-0-138-224 kubenswrapper[2579]: E0909 20:52:40.285683    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gz9zx" podUID="732d9d71-d680-4291-aaee-028307ef79dc"
Sep 09 20:52:41.288679 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:41.288650    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b6qr7"
Sep 09 20:52:41.289158 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:41.288650    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ws8hc"
Sep 09 20:52:41.289158 ip-10-0-138-224 kubenswrapper[2579]: E0909 20:52:41.288783    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b6qr7" podUID="e1193e63-dea9-41f4-a6ab-748d8ae109e8"
Sep 09 20:52:41.289158 ip-10-0-138-224 kubenswrapper[2579]: E0909 20:52:41.288843    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ws8hc" podUID="c6fd107c-ea2f-4e85-9a32-1e11f0858fd3"
Sep 09 20:52:42.285966 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:42.285936    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gz9zx"
Sep 09 20:52:42.286163 ip-10-0-138-224 kubenswrapper[2579]: E0909 20:52:42.286065    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gz9zx" podUID="732d9d71-d680-4291-aaee-028307ef79dc"
Sep 09 20:52:43.288916 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:43.288869    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b6qr7"
Sep 09 20:52:43.289327 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:43.289125    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ws8hc"
Sep 09 20:52:43.289327 ip-10-0-138-224 kubenswrapper[2579]: E0909 20:52:43.289226    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b6qr7" podUID="e1193e63-dea9-41f4-a6ab-748d8ae109e8"
Sep 09 20:52:43.289432 ip-10-0-138-224 kubenswrapper[2579]: E0909 20:52:43.289355    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ws8hc" podUID="c6fd107c-ea2f-4e85-9a32-1e11f0858fd3"
Sep 09 20:52:44.285607 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:44.285568    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gz9zx"
Sep 09 20:52:44.285786 ip-10-0-138-224 kubenswrapper[2579]: E0909 20:52:44.285694    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gz9zx" podUID="732d9d71-d680-4291-aaee-028307ef79dc"
Sep 09 20:52:45.285498 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:45.285177    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b6qr7"
Sep 09 20:52:45.286087 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:45.285236    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ws8hc"
Sep 09 20:52:45.286087 ip-10-0-138-224 kubenswrapper[2579]: E0909 20:52:45.285596    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b6qr7" podUID="e1193e63-dea9-41f4-a6ab-748d8ae109e8"
Sep 09 20:52:45.286087 ip-10-0-138-224 kubenswrapper[2579]: E0909 20:52:45.285656    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ws8hc" podUID="c6fd107c-ea2f-4e85-9a32-1e11f0858fd3"
Sep 09 20:52:45.383814 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:45.383782    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-jbs9t" event={"ID":"22f66b47-36ad-4226-b0f9-ad4075eaaa2d","Type":"ContainerStarted","Data":"8169e3f73c7409bd99e8f1cff154ec7aa3d79e9e29745f73ef0afc768029cbed"}
Sep 09 20:52:45.385328 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:45.385299    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-kwvw7" event={"ID":"9075ffbd-9633-49ac-b6aa-f410776c4ad6","Type":"ContainerStarted","Data":"f7b274dc3ef34a4b5675650a68df6e8faa03f96d0830cb9982e31d38516ff040"}
Sep 09 20:52:45.386815 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:45.386792    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pdpk7" event={"ID":"a55d0249-76a1-42d0-8e06-8c4fa1e557f8","Type":"ContainerStarted","Data":"aab33eefeff677d1a60d94186b9135f038551c99627bd7805a926cc3a2b80215"}
Sep 09 20:52:45.388172 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:45.388150    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-tml69" event={"ID":"169cade8-7395-4b62-9544-ff0c47022a89","Type":"ContainerStarted","Data":"f4907737057b85b2e48bc28cfbd7db090e56a2ba90d45359b24a610ae0fdcfa8"}
Sep 09 20:52:45.390811 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:45.390700    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9dcqf" event={"ID":"b2481c54-52e5-44b3-a24a-09b4dd9f575b","Type":"ContainerStarted","Data":"071fd89033119df1fae2476c0bb4b18ba4c9523ca483ef3caf9f6c32d61b96cc"}
Sep 09 20:52:45.390811 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:45.390750    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9dcqf" event={"ID":"b2481c54-52e5-44b3-a24a-09b4dd9f575b","Type":"ContainerStarted","Data":"61f1142acc614bcfda62d40e6d99a638f98ab3e6aed6e7b0ccb945ae1469621c"}
Sep 09 20:52:45.390811 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:45.390793    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9dcqf" event={"ID":"b2481c54-52e5-44b3-a24a-09b4dd9f575b","Type":"ContainerStarted","Data":"a3cc6b317b9c76eb0d38c94b66fb615cbe47d8626bcc4814e72c3d95e982cc67"}
Sep 09 20:52:45.390976 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:45.390831    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9dcqf" event={"ID":"b2481c54-52e5-44b3-a24a-09b4dd9f575b","Type":"ContainerStarted","Data":"65dc21ba548d09c69c74c22333e6b24d0557ac3b9d4d164c39d744abace3ab4b"}
Sep 09 20:52:45.390976 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:45.390865    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9dcqf" event={"ID":"b2481c54-52e5-44b3-a24a-09b4dd9f575b","Type":"ContainerStarted","Data":"1d570c3cff35c8694f3ebd762bfa50e597f6967c8b7133f73813cb3cbebd9329"}
Sep 09 20:52:45.392087 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:45.392067    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-s98q4" event={"ID":"10f78cd8-5259-4024-ae5f-68bfa6a0626f","Type":"ContainerStarted","Data":"6500178aed70084e83ba432f6572ac806c276200935d91159a429d4ca93be04f"}
Sep 09 20:52:45.393359 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:45.393341    2579 generic.go:356] "Generic (PLEG): container finished" podID="dfe1a979-9dff-419b-b517-93e03c19c040" containerID="3ed5c280b725f95b33e1eba0b8ef71a8fdae3200e3313cde3c9fe28bbc2d97d4" exitCode=0
Sep 09 20:52:45.393457 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:45.393364    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9rprp" event={"ID":"dfe1a979-9dff-419b-b517-93e03c19c040","Type":"ContainerDied","Data":"3ed5c280b725f95b33e1eba0b8ef71a8fdae3200e3313cde3c9fe28bbc2d97d4"}
Sep 09 20:52:45.394740 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:45.394719    2579 generic.go:356] "Generic (PLEG): container finished" podID="a125a7c81ad75870351cc3dc16d80838" containerID="db7e2aecc60faca3f53c85b61022fd0bb9054a0622fbb0ade5760e179ba58e60" exitCode=0
Sep 09 20:52:45.394817 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:45.394788    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-224.ec2.internal" event={"ID":"a125a7c81ad75870351cc3dc16d80838","Type":"ContainerDied","Data":"db7e2aecc60faca3f53c85b61022fd0bb9054a0622fbb0ade5760e179ba58e60"}
Sep 09 20:52:45.396225 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:45.396207    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-88pfs" event={"ID":"ddb26243-140d-4ac9-8a3d-4079559bd0e7","Type":"ContainerStarted","Data":"a60071536f388e3139331ac25eca8de5cc4646e1a5de811f45137f5b09566eb8"}
Sep 09 20:52:45.397649 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:45.397626    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-224.ec2.internal" event={"ID":"7e4d14d2e217a05b09faab3ef55f4d0f","Type":"ContainerStarted","Data":"5dc90d7fcc140d054ad554ed635e01cab8710d9681dd7b4045606ec66595f830"}
Sep 09 20:52:45.400718 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:45.400678    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-jbs9t" podStartSLOduration=2.593691529 podStartE2EDuration="22.400664367s" totalImagesPullingTime="19.806972838s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:52:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:52:45.400548557 +0000 UTC m=+22.640642676" watchObservedRunningTime="2026-09-09 20:52:45.400664367 +0000 UTC m=+22.640758487"
Sep 09 20:52:45.414892 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:45.414855    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-88pfs" podStartSLOduration=2.771746277 podStartE2EDuration="22.414844359s" totalImagesPullingTime="19.643098082s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:52:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:52:45.414564765 +0000 UTC m=+22.654658884" watchObservedRunningTime="2026-09-09 20:52:45.414844359 +0000 UTC m=+22.654938477"
Sep 09 20:52:45.457952 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:45.457913    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-s98q4" podStartSLOduration=2.729416408 podStartE2EDuration="22.457897675s" totalImagesPullingTime="19.728481267s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:52:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:52:45.457644737 +0000 UTC m=+22.697738867" watchObservedRunningTime="2026-09-09 20:52:45.457897675 +0000 UTC m=+22.697991793"
Sep 09 20:52:45.492993 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:45.492935    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-kwvw7" podStartSLOduration=2.832100284 podStartE2EDuration="22.492916234s" totalImagesPullingTime="19.66081595s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:52:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:52:45.492586904 +0000 UTC m=+22.732681023" watchObservedRunningTime="2026-09-09 20:52:45.492916234 +0000 UTC m=+22.733010353"
Sep 09 20:52:45.495473 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:45.494046    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-tml69" podStartSLOduration=2.7480927939999997 podStartE2EDuration="22.494028587s" totalImagesPullingTime="19.745935793s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:52:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:52:45.475210849 +0000 UTC m=+22.715304966" watchObservedRunningTime="2026-09-09 20:52:45.494028587 +0000 UTC m=+22.734122706"
Sep 09 20:52:45.510756 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:45.510718    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-224.ec2.internal" podStartSLOduration=22.510704234 podStartE2EDuration="22.510704234s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:52:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:52:45.510540693 +0000 UTC m=+22.750634812" watchObservedRunningTime="2026-09-09 20:52:45.510704234 +0000 UTC m=+22.750798351"
Sep 09 20:52:46.285984 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:46.285954    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gz9zx"
Sep 09 20:52:46.286586 ip-10-0-138-224 kubenswrapper[2579]: E0909 20:52:46.286060    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gz9zx" podUID="732d9d71-d680-4291-aaee-028307ef79dc"
Sep 09 20:52:46.399564 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:46.399488    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-kwvw7"
Sep 09 20:52:46.400138 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:46.400114    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-kwvw7"
Sep 09 20:52:46.403779 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:46.403755    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9dcqf" event={"ID":"b2481c54-52e5-44b3-a24a-09b4dd9f575b","Type":"ContainerStarted","Data":"1c860587d090eeb6a97e69cf9eadef2f1bef078f43db36029239b0d75e868bbc"}
Sep 09 20:52:46.405374 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:46.405351    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-kfpbw" event={"ID":"28da7add-257c-49f6-a846-6e8116b68d4c","Type":"ContainerStarted","Data":"4cd880ef940d09af2bbe6761edb24cf7aba9a81e272b6f576391174121278604"}
Sep 09 20:52:46.406743 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:46.406725    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-kwvw7"
Sep 09 20:52:46.407286 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:46.406998    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-kwvw7"
Sep 09 20:52:46.430549 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:46.430504    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-kfpbw" podStartSLOduration=4.552336243 podStartE2EDuration="23.430490667s" totalImagesPullingTime="18.878154424s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:52:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:52:46.430080341 +0000 UTC m=+23.670174462" watchObservedRunningTime="2026-09-09 20:52:46.430490667 +0000 UTC m=+23.670584785"
Sep 09 20:52:46.728712 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:46.728687    2579 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 09 20:52:47.228972 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:47.228867    2579 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-09T20:52:46.728706209Z","UUID":"1cd27ada-ffb2-4232-ac49-ab0e3b77737c","Handler":null,"Name":"","Endpoint":""}
Sep 09 20:52:47.230746 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:47.230728    2579 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 09 20:52:47.230864 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:47.230762    2579 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 09 20:52:47.285376 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:47.285347    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ws8hc"
Sep 09 20:52:47.285376 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:47.285367    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b6qr7"
Sep 09 20:52:47.285631 ip-10-0-138-224 kubenswrapper[2579]: E0909 20:52:47.285508    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ws8hc" podUID="c6fd107c-ea2f-4e85-9a32-1e11f0858fd3"
Sep 09 20:52:47.285687 ip-10-0-138-224 kubenswrapper[2579]: E0909 20:52:47.285641    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b6qr7" podUID="e1193e63-dea9-41f4-a6ab-748d8ae109e8"
Sep 09 20:52:47.413005 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:47.412964    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pdpk7" event={"ID":"a55d0249-76a1-42d0-8e06-8c4fa1e557f8","Type":"ContainerStarted","Data":"bbb1163c2983d947480fd56ab599ebd8122895b633ca88b8a0be2d263307e0d3"}
Sep 09 20:52:47.415023 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:47.414991    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-224.ec2.internal" event={"ID":"a125a7c81ad75870351cc3dc16d80838","Type":"ContainerStarted","Data":"0cc462e191dc4a0e1afe96fc4823754f460ba39d878ba5ae6b68e5ce67a745b6"}
Sep 09 20:52:47.436403 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:47.436357    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-224.ec2.internal" podStartSLOduration=24.43634046 podStartE2EDuration="24.43634046s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:52:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:52:47.43561114 +0000 UTC m=+24.675705259" watchObservedRunningTime="2026-09-09 20:52:47.43634046 +0000 UTC m=+24.676434578"
Sep 09 20:52:48.285189 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:48.285005    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gz9zx"
Sep 09 20:52:48.285367 ip-10-0-138-224 kubenswrapper[2579]: E0909 20:52:48.285277    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gz9zx" podUID="732d9d71-d680-4291-aaee-028307ef79dc"
Sep 09 20:52:48.421159 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:48.421111    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9dcqf" event={"ID":"b2481c54-52e5-44b3-a24a-09b4dd9f575b","Type":"ContainerStarted","Data":"c90af7f2060a017006c8cb051028e5a64ea669ac950fef9cb0ca1ba9fac5e248"}
Sep 09 20:52:49.285425 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:49.285389    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b6qr7"
Sep 09 20:52:49.285650 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:49.285397    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ws8hc"
Sep 09 20:52:49.285650 ip-10-0-138-224 kubenswrapper[2579]: E0909 20:52:49.285548    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b6qr7" podUID="e1193e63-dea9-41f4-a6ab-748d8ae109e8"
Sep 09 20:52:49.285752 ip-10-0-138-224 kubenswrapper[2579]: E0909 20:52:49.285681    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ws8hc" podUID="c6fd107c-ea2f-4e85-9a32-1e11f0858fd3"
Sep 09 20:52:50.285747 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:50.285546    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gz9zx"
Sep 09 20:52:50.286111 ip-10-0-138-224 kubenswrapper[2579]: E0909 20:52:50.285811    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gz9zx" podUID="732d9d71-d680-4291-aaee-028307ef79dc"
Sep 09 20:52:50.429222 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:50.429193    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9dcqf" event={"ID":"b2481c54-52e5-44b3-a24a-09b4dd9f575b","Type":"ContainerStarted","Data":"43e240ef7bcd0222146b75681db4162b895b4287baff923b161edfeb4410c7be"}
Sep 09 20:52:50.429485 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:50.429464    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-9dcqf"
Sep 09 20:52:50.431301 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:50.431280    2579 generic.go:356] "Generic (PLEG): container finished" podID="dfe1a979-9dff-419b-b517-93e03c19c040" containerID="7c0b1158d16b1d8ffc256bddd7cc47822e9c6aecf7863cd07a54dfffeca8e6c5" exitCode=0
Sep 09 20:52:50.431400 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:50.431352    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9rprp" event={"ID":"dfe1a979-9dff-419b-b517-93e03c19c040","Type":"ContainerDied","Data":"7c0b1158d16b1d8ffc256bddd7cc47822e9c6aecf7863cd07a54dfffeca8e6c5"}
Sep 09 20:52:50.433822 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:50.433800    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pdpk7" event={"ID":"a55d0249-76a1-42d0-8e06-8c4fa1e557f8","Type":"ContainerStarted","Data":"441e08c4647572a25c91154c5f8eae83c254a283a1d979bb7634a0b93e2e42f6"}
Sep 09 20:52:50.445468 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:50.445450    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-9dcqf"
Sep 09 20:52:50.465148 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:50.465110    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-9dcqf" podStartSLOduration=7.680582827 podStartE2EDuration="27.465097971s" totalImagesPullingTime="19.784515144s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:52:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:52:50.463968222 +0000 UTC m=+27.704062339" watchObservedRunningTime="2026-09-09 20:52:50.465097971 +0000 UTC m=+27.705192086"
Sep 09 20:52:50.533000 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:50.532962    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pdpk7" podStartSLOduration=3.439117669 podStartE2EDuration="27.532949585s" totalImagesPullingTime="24.093831916s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:52:23 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:52:50.532598324 +0000 UTC m=+27.772692441" watchObservedRunningTime="2026-09-09 20:52:50.532949585 +0000 UTC m=+27.773043703"
Sep 09 20:52:51.286068 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:51.286034    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ws8hc"
Sep 09 20:52:51.286521 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:51.286041    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b6qr7"
Sep 09 20:52:51.286521 ip-10-0-138-224 kubenswrapper[2579]: E0909 20:52:51.286183    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ws8hc" podUID="c6fd107c-ea2f-4e85-9a32-1e11f0858fd3"
Sep 09 20:52:51.286521 ip-10-0-138-224 kubenswrapper[2579]: E0909 20:52:51.286198    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b6qr7" podUID="e1193e63-dea9-41f4-a6ab-748d8ae109e8"
Sep 09 20:52:51.436200 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:51.436171    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-9dcqf"
Sep 09 20:52:51.436408 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:51.436229    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-9dcqf"
Sep 09 20:52:51.451405 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:51.451383    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-9dcqf"
Sep 09 20:52:52.040963 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:52.040876    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-gz9zx"]
Sep 09 20:52:52.041086 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:52.041055    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gz9zx"
Sep 09 20:52:52.041165 ip-10-0-138-224 kubenswrapper[2579]: E0909 20:52:52.041147    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gz9zx" podUID="732d9d71-d680-4291-aaee-028307ef79dc"
Sep 09 20:52:52.043726 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:52.043703    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-b6qr7"]
Sep 09 20:52:52.043847 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:52.043803    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b6qr7"
Sep 09 20:52:52.043915 ip-10-0-138-224 kubenswrapper[2579]: E0909 20:52:52.043895    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b6qr7" podUID="e1193e63-dea9-41f4-a6ab-748d8ae109e8"
Sep 09 20:52:52.054389 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:52.054365    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-ws8hc"]
Sep 09 20:52:52.054555 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:52.054541    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ws8hc"
Sep 09 20:52:52.054678 ip-10-0-138-224 kubenswrapper[2579]: E0909 20:52:52.054650    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ws8hc" podUID="c6fd107c-ea2f-4e85-9a32-1e11f0858fd3"
Sep 09 20:52:52.439450 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:52.439403    2579 generic.go:356] "Generic (PLEG): container finished" podID="dfe1a979-9dff-419b-b517-93e03c19c040" containerID="5686655c9e0e8fb9485562d02789768baeee0fb06f860c3983ceed0e30f95948" exitCode=0
Sep 09 20:52:52.440184 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:52.439471    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9rprp" event={"ID":"dfe1a979-9dff-419b-b517-93e03c19c040","Type":"ContainerDied","Data":"5686655c9e0e8fb9485562d02789768baeee0fb06f860c3983ceed0e30f95948"}
Sep 09 20:52:53.287021 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:53.286823    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gz9zx"
Sep 09 20:52:53.287169 ip-10-0-138-224 kubenswrapper[2579]: E0909 20:52:53.287110    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gz9zx" podUID="732d9d71-d680-4291-aaee-028307ef79dc"
Sep 09 20:52:54.285152 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:54.285085    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ws8hc"
Sep 09 20:52:54.285499 ip-10-0-138-224 kubenswrapper[2579]: E0909 20:52:54.285183    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ws8hc" podUID="c6fd107c-ea2f-4e85-9a32-1e11f0858fd3"
Sep 09 20:52:54.285499 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:54.285237    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b6qr7"
Sep 09 20:52:54.285499 ip-10-0-138-224 kubenswrapper[2579]: E0909 20:52:54.285317    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b6qr7" podUID="e1193e63-dea9-41f4-a6ab-748d8ae109e8"
Sep 09 20:52:54.446059 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:54.446028    2579 generic.go:356] "Generic (PLEG): container finished" podID="dfe1a979-9dff-419b-b517-93e03c19c040" containerID="737838a338ac3070d5e4a2aa6151ae57c8e9d5e0f48562d0061ef1ae2438193e" exitCode=0
Sep 09 20:52:54.446190 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:54.446092    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9rprp" event={"ID":"dfe1a979-9dff-419b-b517-93e03c19c040","Type":"ContainerDied","Data":"737838a338ac3070d5e4a2aa6151ae57c8e9d5e0f48562d0061ef1ae2438193e"}
Sep 09 20:52:55.286081 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:55.286048    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gz9zx"
Sep 09 20:52:55.286574 ip-10-0-138-224 kubenswrapper[2579]: E0909 20:52:55.286189    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gz9zx" podUID="732d9d71-d680-4291-aaee-028307ef79dc"
Sep 09 20:52:56.013743 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:56.013710    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c6fd107c-ea2f-4e85-9a32-1e11f0858fd3-metrics-certs\") pod \"network-metrics-daemon-ws8hc\" (UID: \"c6fd107c-ea2f-4e85-9a32-1e11f0858fd3\") " pod="openshift-multus/network-metrics-daemon-ws8hc"
Sep 09 20:52:56.013910 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:56.013764    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/732d9d71-d680-4291-aaee-028307ef79dc-original-pull-secret\") pod \"global-pull-secret-syncer-gz9zx\" (UID: \"732d9d71-d680-4291-aaee-028307ef79dc\") " pod="kube-system/global-pull-secret-syncer-gz9zx"
Sep 09 20:52:56.013910 ip-10-0-138-224 kubenswrapper[2579]: E0909 20:52:56.013871    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 20:52:56.013910 ip-10-0-138-224 kubenswrapper[2579]: E0909 20:52:56.013874    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:52:56.014041 ip-10-0-138-224 kubenswrapper[2579]: E0909 20:52:56.013936    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/732d9d71-d680-4291-aaee-028307ef79dc-original-pull-secret podName:732d9d71-d680-4291-aaee-028307ef79dc nodeName:}" failed. No retries permitted until 2026-09-09 20:53:28.013918152 +0000 UTC m=+65.254012262 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/732d9d71-d680-4291-aaee-028307ef79dc-original-pull-secret") pod "global-pull-secret-syncer-gz9zx" (UID: "732d9d71-d680-4291-aaee-028307ef79dc") : object "kube-system"/"original-pull-secret" not registered
Sep 09 20:52:56.014041 ip-10-0-138-224 kubenswrapper[2579]: E0909 20:52:56.013982    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c6fd107c-ea2f-4e85-9a32-1e11f0858fd3-metrics-certs podName:c6fd107c-ea2f-4e85-9a32-1e11f0858fd3 nodeName:}" failed. No retries permitted until 2026-09-09 20:53:28.013961012 +0000 UTC m=+65.254055124 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c6fd107c-ea2f-4e85-9a32-1e11f0858fd3-metrics-certs") pod "network-metrics-daemon-ws8hc" (UID: "c6fd107c-ea2f-4e85-9a32-1e11f0858fd3") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:52:56.114613 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:56.114568    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6c58d\" (UniqueName: \"kubernetes.io/projected/e1193e63-dea9-41f4-a6ab-748d8ae109e8-kube-api-access-6c58d\") pod \"network-check-target-b6qr7\" (UID: \"e1193e63-dea9-41f4-a6ab-748d8ae109e8\") " pod="openshift-network-diagnostics/network-check-target-b6qr7"
Sep 09 20:52:56.114776 ip-10-0-138-224 kubenswrapper[2579]: E0909 20:52:56.114740    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 20:52:56.114776 ip-10-0-138-224 kubenswrapper[2579]: E0909 20:52:56.114768    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 20:52:56.114896 ip-10-0-138-224 kubenswrapper[2579]: E0909 20:52:56.114784    2579 projected.go:196] Error preparing data for projected volume kube-api-access-6c58d for pod openshift-network-diagnostics/network-check-target-b6qr7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:52:56.114896 ip-10-0-138-224 kubenswrapper[2579]: E0909 20:52:56.114841    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e1193e63-dea9-41f4-a6ab-748d8ae109e8-kube-api-access-6c58d podName:e1193e63-dea9-41f4-a6ab-748d8ae109e8 nodeName:}" failed. No retries permitted until 2026-09-09 20:53:28.114823137 +0000 UTC m=+65.354917256 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-6c58d" (UniqueName: "kubernetes.io/projected/e1193e63-dea9-41f4-a6ab-748d8ae109e8-kube-api-access-6c58d") pod "network-check-target-b6qr7" (UID: "e1193e63-dea9-41f4-a6ab-748d8ae109e8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:52:56.285164 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:56.285051    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b6qr7"
Sep 09 20:52:56.285164 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:56.285086    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ws8hc"
Sep 09 20:52:56.285362 ip-10-0-138-224 kubenswrapper[2579]: E0909 20:52:56.285176    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b6qr7" podUID="e1193e63-dea9-41f4-a6ab-748d8ae109e8"
Sep 09 20:52:56.285362 ip-10-0-138-224 kubenswrapper[2579]: E0909 20:52:56.285301    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ws8hc" podUID="c6fd107c-ea2f-4e85-9a32-1e11f0858fd3"
Sep 09 20:52:57.285470 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:57.285414    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gz9zx"
Sep 09 20:52:57.286093 ip-10-0-138-224 kubenswrapper[2579]: E0909 20:52:57.285560    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gz9zx" podUID="732d9d71-d680-4291-aaee-028307ef79dc"
Sep 09 20:52:57.531531 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:57.531495    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-224.ec2.internal" event="NodeReady"
Sep 09 20:52:57.531710 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:57.531642    2579 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 09 20:52:57.590567 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:57.590477    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-zwg8k"]
Sep 09 20:52:57.595455 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:57.595418    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-zwg8k"
Sep 09 20:52:57.597321 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:57.596592    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-h4v54"]
Sep 09 20:52:57.598219 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:57.598197    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 09 20:52:57.598327 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:57.598202    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 09 20:52:57.598327 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:57.598202    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-7vdcc\""
Sep 09 20:52:57.600193 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:57.600172    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-h4v54"
Sep 09 20:52:57.602853 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:57.602700    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 09 20:52:57.602853 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:57.602801    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 09 20:52:57.603774 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:57.603753    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-f6qjq\""
Sep 09 20:52:57.603874 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:57.603774    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 09 20:52:57.609925 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:57.609908    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-zwg8k"]
Sep 09 20:52:57.612759 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:57.612737    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-h4v54"]
Sep 09 20:52:57.727179 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:57.727138    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8de0dbc6-534d-4a25-b218-8e13816e7932-tmp-dir\") pod \"dns-default-zwg8k\" (UID: \"8de0dbc6-534d-4a25-b218-8e13816e7932\") " pod="openshift-dns/dns-default-zwg8k"
Sep 09 20:52:57.727179 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:57.727181    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8de0dbc6-534d-4a25-b218-8e13816e7932-metrics-tls\") pod \"dns-default-zwg8k\" (UID: \"8de0dbc6-534d-4a25-b218-8e13816e7932\") " pod="openshift-dns/dns-default-zwg8k"
Sep 09 20:52:57.727400 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:57.727199    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89w6l\" (UniqueName: \"kubernetes.io/projected/5555e03e-a908-4628-a153-a8e0f4bf3f42-kube-api-access-89w6l\") pod \"ingress-canary-h4v54\" (UID: \"5555e03e-a908-4628-a153-a8e0f4bf3f42\") " pod="openshift-ingress-canary/ingress-canary-h4v54"
Sep 09 20:52:57.727400 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:57.727271    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8de0dbc6-534d-4a25-b218-8e13816e7932-config-volume\") pod \"dns-default-zwg8k\" (UID: \"8de0dbc6-534d-4a25-b218-8e13816e7932\") " pod="openshift-dns/dns-default-zwg8k"
Sep 09 20:52:57.727400 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:57.727293    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5555e03e-a908-4628-a153-a8e0f4bf3f42-cert\") pod \"ingress-canary-h4v54\" (UID: \"5555e03e-a908-4628-a153-a8e0f4bf3f42\") " pod="openshift-ingress-canary/ingress-canary-h4v54"
Sep 09 20:52:57.727400 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:57.727318    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bc57f\" (UniqueName: \"kubernetes.io/projected/8de0dbc6-534d-4a25-b218-8e13816e7932-kube-api-access-bc57f\") pod \"dns-default-zwg8k\" (UID: \"8de0dbc6-534d-4a25-b218-8e13816e7932\") " pod="openshift-dns/dns-default-zwg8k"
Sep 09 20:52:57.828162 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:57.828123    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bc57f\" (UniqueName: \"kubernetes.io/projected/8de0dbc6-534d-4a25-b218-8e13816e7932-kube-api-access-bc57f\") pod \"dns-default-zwg8k\" (UID: \"8de0dbc6-534d-4a25-b218-8e13816e7932\") " pod="openshift-dns/dns-default-zwg8k"
Sep 09 20:52:57.828337 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:57.828209    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8de0dbc6-534d-4a25-b218-8e13816e7932-tmp-dir\") pod \"dns-default-zwg8k\" (UID: \"8de0dbc6-534d-4a25-b218-8e13816e7932\") " pod="openshift-dns/dns-default-zwg8k"
Sep 09 20:52:57.828337 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:57.828238    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8de0dbc6-534d-4a25-b218-8e13816e7932-metrics-tls\") pod \"dns-default-zwg8k\" (UID: \"8de0dbc6-534d-4a25-b218-8e13816e7932\") " pod="openshift-dns/dns-default-zwg8k"
Sep 09 20:52:57.828337 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:57.828262    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89w6l\" (UniqueName: \"kubernetes.io/projected/5555e03e-a908-4628-a153-a8e0f4bf3f42-kube-api-access-89w6l\") pod \"ingress-canary-h4v54\" (UID: \"5555e03e-a908-4628-a153-a8e0f4bf3f42\") " pod="openshift-ingress-canary/ingress-canary-h4v54"
Sep 09 20:52:57.828337 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:57.828317    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8de0dbc6-534d-4a25-b218-8e13816e7932-config-volume\") pod \"dns-default-zwg8k\" (UID: \"8de0dbc6-534d-4a25-b218-8e13816e7932\") " pod="openshift-dns/dns-default-zwg8k"
Sep 09 20:52:57.828601 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:57.828344    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5555e03e-a908-4628-a153-a8e0f4bf3f42-cert\") pod \"ingress-canary-h4v54\" (UID: \"5555e03e-a908-4628-a153-a8e0f4bf3f42\") " pod="openshift-ingress-canary/ingress-canary-h4v54"
Sep 09 20:52:57.828601 ip-10-0-138-224 kubenswrapper[2579]: E0909 20:52:57.828403    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 20:52:57.828601 ip-10-0-138-224 kubenswrapper[2579]: E0909 20:52:57.828493    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8de0dbc6-534d-4a25-b218-8e13816e7932-metrics-tls podName:8de0dbc6-534d-4a25-b218-8e13816e7932 nodeName:}" failed. No retries permitted until 2026-09-09 20:52:58.32847191 +0000 UTC m=+35.568566021 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8de0dbc6-534d-4a25-b218-8e13816e7932-metrics-tls") pod "dns-default-zwg8k" (UID: "8de0dbc6-534d-4a25-b218-8e13816e7932") : secret "dns-default-metrics-tls" not found
Sep 09 20:52:57.828788 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:57.828627    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8de0dbc6-534d-4a25-b218-8e13816e7932-tmp-dir\") pod \"dns-default-zwg8k\" (UID: \"8de0dbc6-534d-4a25-b218-8e13816e7932\") " pod="openshift-dns/dns-default-zwg8k"
Sep 09 20:52:57.829094 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:57.829075    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8de0dbc6-534d-4a25-b218-8e13816e7932-config-volume\") pod \"dns-default-zwg8k\" (UID: \"8de0dbc6-534d-4a25-b218-8e13816e7932\") " pod="openshift-dns/dns-default-zwg8k"
Sep 09 20:52:57.833130 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:57.833111    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5555e03e-a908-4628-a153-a8e0f4bf3f42-cert\") pod \"ingress-canary-h4v54\" (UID: \"5555e03e-a908-4628-a153-a8e0f4bf3f42\") " pod="openshift-ingress-canary/ingress-canary-h4v54"
Sep 09 20:52:57.841566 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:57.841513    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bc57f\" (UniqueName: \"kubernetes.io/projected/8de0dbc6-534d-4a25-b218-8e13816e7932-kube-api-access-bc57f\") pod \"dns-default-zwg8k\" (UID: \"8de0dbc6-534d-4a25-b218-8e13816e7932\") " pod="openshift-dns/dns-default-zwg8k"
Sep 09 20:52:57.841998 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:57.841971    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-89w6l\" (UniqueName: \"kubernetes.io/projected/5555e03e-a908-4628-a153-a8e0f4bf3f42-kube-api-access-89w6l\") pod \"ingress-canary-h4v54\" (UID: \"5555e03e-a908-4628-a153-a8e0f4bf3f42\") " pod="openshift-ingress-canary/ingress-canary-h4v54"
Sep 09 20:52:57.913765 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:57.913730    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-h4v54"
Sep 09 20:52:57.921408 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:57.921382    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-bm5qc"]
Sep 09 20:52:57.924878 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:57.924855    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-bm5qc"
Sep 09 20:52:57.928305 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:57.928284    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-6jl7l\""
Sep 09 20:52:57.928989 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:57.928970    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Sep 09 20:52:57.929102 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:57.928973    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Sep 09 20:52:57.945391 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:57.945359    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-bm5qc"]
Sep 09 20:52:58.013054 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:58.013018    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-64bb684d65-d9qxp"]
Sep 09 20:52:58.016751 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:58.016724    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-64bb684d65-d9qxp"
Sep 09 20:52:58.020227 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:58.020198    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 09 20:52:58.020756 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:58.020419    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 09 20:52:58.020756 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:58.020508    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-lr6q6\""
Sep 09 20:52:58.020756 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:58.020631    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 09 20:52:58.027968 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:58.027947    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 09 20:52:58.029219 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:58.029193    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nl8w6\" (UniqueName: \"kubernetes.io/projected/c31fd8ed-0c33-446d-9b0e-76fdd5935030-kube-api-access-nl8w6\") pod \"volume-data-source-validator-695c895bd6-bm5qc\" (UID: \"c31fd8ed-0c33-446d-9b0e-76fdd5935030\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-bm5qc"
Sep 09 20:52:58.039169 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:58.039142    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-64bb684d65-d9qxp"]
Sep 09 20:52:58.102967 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:58.102879    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cdcc96965-gqh2c"]
Sep 09 20:52:58.106656 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:58.106631    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-gqh2c"
Sep 09 20:52:58.110471 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:58.110348    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Sep 09 20:52:58.110840 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:58.110503    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Sep 09 20:52:58.110840 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:58.110571    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 09 20:52:58.111732 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:58.111703    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-vd94f\""
Sep 09 20:52:58.111850 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:58.111742    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 09 20:52:58.123492 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:58.123464    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cdcc96965-gqh2c"]
Sep 09 20:52:58.130070 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:58.130041    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/729f090c-e6f4-4757-98ec-9eba867c9ca8-ca-trust-extracted\") pod \"image-registry-64bb684d65-d9qxp\" (UID: \"729f090c-e6f4-4757-98ec-9eba867c9ca8\") " pod="openshift-image-registry/image-registry-64bb684d65-d9qxp"
Sep 09 20:52:58.130177 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:58.130085    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/729f090c-e6f4-4757-98ec-9eba867c9ca8-installation-pull-secrets\") pod \"image-registry-64bb684d65-d9qxp\" (UID: \"729f090c-e6f4-4757-98ec-9eba867c9ca8\") " pod="openshift-image-registry/image-registry-64bb684d65-d9qxp"
Sep 09 20:52:58.130234 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:58.130174    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/729f090c-e6f4-4757-98ec-9eba867c9ca8-bound-sa-token\") pod \"image-registry-64bb684d65-d9qxp\" (UID: \"729f090c-e6f4-4757-98ec-9eba867c9ca8\") " pod="openshift-image-registry/image-registry-64bb684d65-d9qxp"
Sep 09 20:52:58.130302 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:58.130285    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/729f090c-e6f4-4757-98ec-9eba867c9ca8-image-registry-private-configuration\") pod \"image-registry-64bb684d65-d9qxp\" (UID: \"729f090c-e6f4-4757-98ec-9eba867c9ca8\") " pod="openshift-image-registry/image-registry-64bb684d65-d9qxp"
Sep 09 20:52:58.130344 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:58.130311    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/729f090c-e6f4-4757-98ec-9eba867c9ca8-trusted-ca\") pod \"image-registry-64bb684d65-d9qxp\" (UID: \"729f090c-e6f4-4757-98ec-9eba867c9ca8\") " pod="openshift-image-registry/image-registry-64bb684d65-d9qxp"
Sep 09 20:52:58.130389 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:58.130338    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kt8r9\" (UniqueName: \"kubernetes.io/projected/729f090c-e6f4-4757-98ec-9eba867c9ca8-kube-api-access-kt8r9\") pod \"image-registry-64bb684d65-d9qxp\" (UID: \"729f090c-e6f4-4757-98ec-9eba867c9ca8\") " pod="openshift-image-registry/image-registry-64bb684d65-d9qxp"
Sep 09 20:52:58.130434 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:58.130390    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nl8w6\" (UniqueName: \"kubernetes.io/projected/c31fd8ed-0c33-446d-9b0e-76fdd5935030-kube-api-access-nl8w6\") pod \"volume-data-source-validator-695c895bd6-bm5qc\" (UID: \"c31fd8ed-0c33-446d-9b0e-76fdd5935030\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-bm5qc"
Sep 09 20:52:58.130498 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:58.130475    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/729f090c-e6f4-4757-98ec-9eba867c9ca8-registry-tls\") pod \"image-registry-64bb684d65-d9qxp\" (UID: \"729f090c-e6f4-4757-98ec-9eba867c9ca8\") " pod="openshift-image-registry/image-registry-64bb684d65-d9qxp"
Sep 09 20:52:58.130576 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:58.130561    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/729f090c-e6f4-4757-98ec-9eba867c9ca8-registry-certificates\") pod \"image-registry-64bb684d65-d9qxp\" (UID: \"729f090c-e6f4-4757-98ec-9eba867c9ca8\") " pod="openshift-image-registry/image-registry-64bb684d65-d9qxp"
Sep 09 20:52:58.144451 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:58.144400    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nl8w6\" (UniqueName: \"kubernetes.io/projected/c31fd8ed-0c33-446d-9b0e-76fdd5935030-kube-api-access-nl8w6\") pod \"volume-data-source-validator-695c895bd6-bm5qc\" (UID: \"c31fd8ed-0c33-446d-9b0e-76fdd5935030\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-bm5qc"
Sep 09 20:52:58.231301 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:58.231265    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/729f090c-e6f4-4757-98ec-9eba867c9ca8-registry-certificates\") pod \"image-registry-64bb684d65-d9qxp\" (UID: \"729f090c-e6f4-4757-98ec-9eba867c9ca8\") " pod="openshift-image-registry/image-registry-64bb684d65-d9qxp"
Sep 09 20:52:58.231521 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:58.231315    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d7f530ee-5bfc-4e22-acc2-328625228733-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-gqh2c\" (UID: \"d7f530ee-5bfc-4e22-acc2-328625228733\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-gqh2c"
Sep 09 20:52:58.231521 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:58.231344    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/d7f530ee-5bfc-4e22-acc2-328625228733-telemetry-config\") pod \"cluster-monitoring-operator-6cdcc96965-gqh2c\" (UID: \"d7f530ee-5bfc-4e22-acc2-328625228733\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-gqh2c"
Sep 09 20:52:58.231521 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:58.231371    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/729f090c-e6f4-4757-98ec-9eba867c9ca8-ca-trust-extracted\") pod \"image-registry-64bb684d65-d9qxp\" (UID: \"729f090c-e6f4-4757-98ec-9eba867c9ca8\") " pod="openshift-image-registry/image-registry-64bb684d65-d9qxp"
Sep 09 20:52:58.231521 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:58.231394    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/729f090c-e6f4-4757-98ec-9eba867c9ca8-installation-pull-secrets\") pod \"image-registry-64bb684d65-d9qxp\" (UID: \"729f090c-e6f4-4757-98ec-9eba867c9ca8\") " pod="openshift-image-registry/image-registry-64bb684d65-d9qxp"
Sep 09 20:52:58.231521 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:58.231427    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ffr6x\" (UniqueName: \"kubernetes.io/projected/d7f530ee-5bfc-4e22-acc2-328625228733-kube-api-access-ffr6x\") pod \"cluster-monitoring-operator-6cdcc96965-gqh2c\" (UID: \"d7f530ee-5bfc-4e22-acc2-328625228733\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-gqh2c"
Sep 09 20:52:58.231521 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:58.231471    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/729f090c-e6f4-4757-98ec-9eba867c9ca8-bound-sa-token\") pod \"image-registry-64bb684d65-d9qxp\" (UID: \"729f090c-e6f4-4757-98ec-9eba867c9ca8\") " pod="openshift-image-registry/image-registry-64bb684d65-d9qxp"
Sep 09 20:52:58.231834 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:58.231526    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/729f090c-e6f4-4757-98ec-9eba867c9ca8-image-registry-private-configuration\") pod \"image-registry-64bb684d65-d9qxp\" (UID: \"729f090c-e6f4-4757-98ec-9eba867c9ca8\") " pod="openshift-image-registry/image-registry-64bb684d65-d9qxp"
Sep 09 20:52:58.231834 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:58.231548    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/729f090c-e6f4-4757-98ec-9eba867c9ca8-trusted-ca\") pod \"image-registry-64bb684d65-d9qxp\" (UID: \"729f090c-e6f4-4757-98ec-9eba867c9ca8\") " pod="openshift-image-registry/image-registry-64bb684d65-d9qxp"
Sep 09 20:52:58.231834 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:58.231573    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kt8r9\" (UniqueName: \"kubernetes.io/projected/729f090c-e6f4-4757-98ec-9eba867c9ca8-kube-api-access-kt8r9\") pod \"image-registry-64bb684d65-d9qxp\" (UID: \"729f090c-e6f4-4757-98ec-9eba867c9ca8\") " pod="openshift-image-registry/image-registry-64bb684d65-d9qxp"
Sep 09 20:52:58.231834 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:58.231627    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/729f090c-e6f4-4757-98ec-9eba867c9ca8-registry-tls\") pod \"image-registry-64bb684d65-d9qxp\" (UID: \"729f090c-e6f4-4757-98ec-9eba867c9ca8\") " pod="openshift-image-registry/image-registry-64bb684d65-d9qxp"
Sep 09 20:52:58.231834 ip-10-0-138-224 kubenswrapper[2579]: E0909 20:52:58.231758    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 09 20:52:58.231834 ip-10-0-138-224 kubenswrapper[2579]: E0909 20:52:58.231773    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-64bb684d65-d9qxp: secret "image-registry-tls" not found
Sep 09 20:52:58.231834 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:58.231772    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/729f090c-e6f4-4757-98ec-9eba867c9ca8-ca-trust-extracted\") pod \"image-registry-64bb684d65-d9qxp\" (UID: \"729f090c-e6f4-4757-98ec-9eba867c9ca8\") " pod="openshift-image-registry/image-registry-64bb684d65-d9qxp"
Sep 09 20:52:58.231834 ip-10-0-138-224 kubenswrapper[2579]: E0909 20:52:58.231828    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/729f090c-e6f4-4757-98ec-9eba867c9ca8-registry-tls podName:729f090c-e6f4-4757-98ec-9eba867c9ca8 nodeName:}" failed. No retries permitted until 2026-09-09 20:52:58.731809711 +0000 UTC m=+35.971903826 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/729f090c-e6f4-4757-98ec-9eba867c9ca8-registry-tls") pod "image-registry-64bb684d65-d9qxp" (UID: "729f090c-e6f4-4757-98ec-9eba867c9ca8") : secret "image-registry-tls" not found
Sep 09 20:52:58.232184 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:58.231959    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/729f090c-e6f4-4757-98ec-9eba867c9ca8-registry-certificates\") pod \"image-registry-64bb684d65-d9qxp\" (UID: \"729f090c-e6f4-4757-98ec-9eba867c9ca8\") " pod="openshift-image-registry/image-registry-64bb684d65-d9qxp"
Sep 09 20:52:58.233073 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:58.233046    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/729f090c-e6f4-4757-98ec-9eba867c9ca8-trusted-ca\") pod \"image-registry-64bb684d65-d9qxp\" (UID: \"729f090c-e6f4-4757-98ec-9eba867c9ca8\") " pod="openshift-image-registry/image-registry-64bb684d65-d9qxp"
Sep 09 20:52:58.234405 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:58.234385    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/729f090c-e6f4-4757-98ec-9eba867c9ca8-image-registry-private-configuration\") pod \"image-registry-64bb684d65-d9qxp\" (UID: \"729f090c-e6f4-4757-98ec-9eba867c9ca8\") " pod="openshift-image-registry/image-registry-64bb684d65-d9qxp"
Sep 09 20:52:58.234530 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:58.234432    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/729f090c-e6f4-4757-98ec-9eba867c9ca8-installation-pull-secrets\") pod \"image-registry-64bb684d65-d9qxp\" (UID: \"729f090c-e6f4-4757-98ec-9eba867c9ca8\") " pod="openshift-image-registry/image-registry-64bb684d65-d9qxp"
Sep 09 20:52:58.234530 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:58.234519    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-bm5qc"
Sep 09 20:52:58.245633 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:58.245608    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kt8r9\" (UniqueName: \"kubernetes.io/projected/729f090c-e6f4-4757-98ec-9eba867c9ca8-kube-api-access-kt8r9\") pod \"image-registry-64bb684d65-d9qxp\" (UID: \"729f090c-e6f4-4757-98ec-9eba867c9ca8\") " pod="openshift-image-registry/image-registry-64bb684d65-d9qxp"
Sep 09 20:52:58.246712 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:58.246695    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/729f090c-e6f4-4757-98ec-9eba867c9ca8-bound-sa-token\") pod \"image-registry-64bb684d65-d9qxp\" (UID: \"729f090c-e6f4-4757-98ec-9eba867c9ca8\") " pod="openshift-image-registry/image-registry-64bb684d65-d9qxp"
Sep 09 20:52:58.285348 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:58.285320    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b6qr7"
Sep 09 20:52:58.285536 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:58.285319    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ws8hc"
Sep 09 20:52:58.288666 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:58.288637    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-w9kf4\""
Sep 09 20:52:58.288989 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:58.288972    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 09 20:52:58.289095 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:58.288973    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 09 20:52:58.289095 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:58.289013    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-7m4wj\""
Sep 09 20:52:58.289522 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:58.289504    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 09 20:52:58.332881 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:58.332845    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8de0dbc6-534d-4a25-b218-8e13816e7932-metrics-tls\") pod \"dns-default-zwg8k\" (UID: \"8de0dbc6-534d-4a25-b218-8e13816e7932\") " pod="openshift-dns/dns-default-zwg8k"
Sep 09 20:52:58.333078 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:58.332935    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d7f530ee-5bfc-4e22-acc2-328625228733-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-gqh2c\" (UID: \"d7f530ee-5bfc-4e22-acc2-328625228733\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-gqh2c"
Sep 09 20:52:58.333078 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:58.332966    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/d7f530ee-5bfc-4e22-acc2-328625228733-telemetry-config\") pod \"cluster-monitoring-operator-6cdcc96965-gqh2c\" (UID: \"d7f530ee-5bfc-4e22-acc2-328625228733\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-gqh2c"
Sep 09 20:52:58.333078 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:58.332998    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffr6x\" (UniqueName: \"kubernetes.io/projected/d7f530ee-5bfc-4e22-acc2-328625228733-kube-api-access-ffr6x\") pod \"cluster-monitoring-operator-6cdcc96965-gqh2c\" (UID: \"d7f530ee-5bfc-4e22-acc2-328625228733\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-gqh2c"
Sep 09 20:52:58.333078 ip-10-0-138-224 kubenswrapper[2579]: E0909 20:52:58.333044    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 20:52:58.333326 ip-10-0-138-224 kubenswrapper[2579]: E0909 20:52:58.333097    2579 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 09 20:52:58.333326 ip-10-0-138-224 kubenswrapper[2579]: E0909 20:52:58.333132    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8de0dbc6-534d-4a25-b218-8e13816e7932-metrics-tls podName:8de0dbc6-534d-4a25-b218-8e13816e7932 nodeName:}" failed. No retries permitted until 2026-09-09 20:52:59.333109805 +0000 UTC m=+36.573203921 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8de0dbc6-534d-4a25-b218-8e13816e7932-metrics-tls") pod "dns-default-zwg8k" (UID: "8de0dbc6-534d-4a25-b218-8e13816e7932") : secret "dns-default-metrics-tls" not found
Sep 09 20:52:58.333326 ip-10-0-138-224 kubenswrapper[2579]: E0909 20:52:58.333166    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7f530ee-5bfc-4e22-acc2-328625228733-cluster-monitoring-operator-tls podName:d7f530ee-5bfc-4e22-acc2-328625228733 nodeName:}" failed. No retries permitted until 2026-09-09 20:52:58.833146546 +0000 UTC m=+36.073240659 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/d7f530ee-5bfc-4e22-acc2-328625228733-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-gqh2c" (UID: "d7f530ee-5bfc-4e22-acc2-328625228733") : secret "cluster-monitoring-operator-tls" not found
Sep 09 20:52:58.333910 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:58.333891    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/d7f530ee-5bfc-4e22-acc2-328625228733-telemetry-config\") pod \"cluster-monitoring-operator-6cdcc96965-gqh2c\" (UID: \"d7f530ee-5bfc-4e22-acc2-328625228733\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-gqh2c"
Sep 09 20:52:58.343816 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:58.343786    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffr6x\" (UniqueName: \"kubernetes.io/projected/d7f530ee-5bfc-4e22-acc2-328625228733-kube-api-access-ffr6x\") pod \"cluster-monitoring-operator-6cdcc96965-gqh2c\" (UID: \"d7f530ee-5bfc-4e22-acc2-328625228733\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-gqh2c"
Sep 09 20:52:58.736520 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:58.736317    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/729f090c-e6f4-4757-98ec-9eba867c9ca8-registry-tls\") pod \"image-registry-64bb684d65-d9qxp\" (UID: \"729f090c-e6f4-4757-98ec-9eba867c9ca8\") " pod="openshift-image-registry/image-registry-64bb684d65-d9qxp"
Sep 09 20:52:58.736635 ip-10-0-138-224 kubenswrapper[2579]: E0909 20:52:58.736484    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 09 20:52:58.736635 ip-10-0-138-224 kubenswrapper[2579]: E0909 20:52:58.736597    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-64bb684d65-d9qxp: secret "image-registry-tls" not found
Sep 09 20:52:58.736710 ip-10-0-138-224 kubenswrapper[2579]: E0909 20:52:58.736654    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/729f090c-e6f4-4757-98ec-9eba867c9ca8-registry-tls podName:729f090c-e6f4-4757-98ec-9eba867c9ca8 nodeName:}" failed. No retries permitted until 2026-09-09 20:52:59.736637474 +0000 UTC m=+36.976731580 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/729f090c-e6f4-4757-98ec-9eba867c9ca8-registry-tls") pod "image-registry-64bb684d65-d9qxp" (UID: "729f090c-e6f4-4757-98ec-9eba867c9ca8") : secret "image-registry-tls" not found
Sep 09 20:52:58.837008 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:58.836926    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d7f530ee-5bfc-4e22-acc2-328625228733-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-gqh2c\" (UID: \"d7f530ee-5bfc-4e22-acc2-328625228733\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-gqh2c"
Sep 09 20:52:58.837151 ip-10-0-138-224 kubenswrapper[2579]: E0909 20:52:58.837061    2579 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 09 20:52:58.837151 ip-10-0-138-224 kubenswrapper[2579]: E0909 20:52:58.837133    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7f530ee-5bfc-4e22-acc2-328625228733-cluster-monitoring-operator-tls podName:d7f530ee-5bfc-4e22-acc2-328625228733 nodeName:}" failed. No retries permitted until 2026-09-09 20:52:59.837113795 +0000 UTC m=+37.077207910 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/d7f530ee-5bfc-4e22-acc2-328625228733-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-gqh2c" (UID: "d7f530ee-5bfc-4e22-acc2-328625228733") : secret "cluster-monitoring-operator-tls" not found
Sep 09 20:52:58.837634 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:58.837610    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-h4v54"]
Sep 09 20:52:58.847381 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:58.847360    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-bm5qc"]
Sep 09 20:52:58.850371 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:58.850337    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc31fd8ed_0c33_446d_9b0e_76fdd5935030.slice/crio-f25079a70eb926c1d3cb386977704331002a5396b304554015bbc933f56c2fe1 WatchSource:0}: Error finding container f25079a70eb926c1d3cb386977704331002a5396b304554015bbc933f56c2fe1: Status 404 returned error can't find the container with id f25079a70eb926c1d3cb386977704331002a5396b304554015bbc933f56c2fe1
Sep 09 20:52:58.851566 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:52:58.851546    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5555e03e_a908_4628_a153_a8e0f4bf3f42.slice/crio-68a6ca3050a8354d1e2c0605a7970879d9b4ff2d41666897ef03b5ae056e6cf2 WatchSource:0}: Error finding container 68a6ca3050a8354d1e2c0605a7970879d9b4ff2d41666897ef03b5ae056e6cf2: Status 404 returned error can't find the container with id 68a6ca3050a8354d1e2c0605a7970879d9b4ff2d41666897ef03b5ae056e6cf2
Sep 09 20:52:59.285378 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:59.285343    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gz9zx"
Sep 09 20:52:59.289749 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:59.289532    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 09 20:52:59.340599 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:59.340571    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8de0dbc6-534d-4a25-b218-8e13816e7932-metrics-tls\") pod \"dns-default-zwg8k\" (UID: \"8de0dbc6-534d-4a25-b218-8e13816e7932\") " pod="openshift-dns/dns-default-zwg8k"
Sep 09 20:52:59.340745 ip-10-0-138-224 kubenswrapper[2579]: E0909 20:52:59.340726    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 20:52:59.340814 ip-10-0-138-224 kubenswrapper[2579]: E0909 20:52:59.340802    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8de0dbc6-534d-4a25-b218-8e13816e7932-metrics-tls podName:8de0dbc6-534d-4a25-b218-8e13816e7932 nodeName:}" failed. No retries permitted until 2026-09-09 20:53:01.340780245 +0000 UTC m=+38.580874364 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8de0dbc6-534d-4a25-b218-8e13816e7932-metrics-tls") pod "dns-default-zwg8k" (UID: "8de0dbc6-534d-4a25-b218-8e13816e7932") : secret "dns-default-metrics-tls" not found
Sep 09 20:52:59.465212 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:59.465177    2579 generic.go:356] "Generic (PLEG): container finished" podID="dfe1a979-9dff-419b-b517-93e03c19c040" containerID="ae74773b4d48f61b08d725bc6909e3970ced5da50ffbf325a174fbbc9b52be0d" exitCode=0
Sep 09 20:52:59.465395 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:59.465252    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9rprp" event={"ID":"dfe1a979-9dff-419b-b517-93e03c19c040","Type":"ContainerDied","Data":"ae74773b4d48f61b08d725bc6909e3970ced5da50ffbf325a174fbbc9b52be0d"}
Sep 09 20:52:59.467206 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:59.467178    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-h4v54" event={"ID":"5555e03e-a908-4628-a153-a8e0f4bf3f42","Type":"ContainerStarted","Data":"68a6ca3050a8354d1e2c0605a7970879d9b4ff2d41666897ef03b5ae056e6cf2"}
Sep 09 20:52:59.468610 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:59.468585    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-bm5qc" event={"ID":"c31fd8ed-0c33-446d-9b0e-76fdd5935030","Type":"ContainerStarted","Data":"f25079a70eb926c1d3cb386977704331002a5396b304554015bbc933f56c2fe1"}
Sep 09 20:52:59.743957 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:59.743893    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/729f090c-e6f4-4757-98ec-9eba867c9ca8-registry-tls\") pod \"image-registry-64bb684d65-d9qxp\" (UID: \"729f090c-e6f4-4757-98ec-9eba867c9ca8\") " pod="openshift-image-registry/image-registry-64bb684d65-d9qxp"
Sep 09 20:52:59.744091 ip-10-0-138-224 kubenswrapper[2579]: E0909 20:52:59.744068    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 09 20:52:59.744091 ip-10-0-138-224 kubenswrapper[2579]: E0909 20:52:59.744084    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-64bb684d65-d9qxp: secret "image-registry-tls" not found
Sep 09 20:52:59.744184 ip-10-0-138-224 kubenswrapper[2579]: E0909 20:52:59.744146    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/729f090c-e6f4-4757-98ec-9eba867c9ca8-registry-tls podName:729f090c-e6f4-4757-98ec-9eba867c9ca8 nodeName:}" failed. No retries permitted until 2026-09-09 20:53:01.744129857 +0000 UTC m=+38.984223973 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/729f090c-e6f4-4757-98ec-9eba867c9ca8-registry-tls") pod "image-registry-64bb684d65-d9qxp" (UID: "729f090c-e6f4-4757-98ec-9eba867c9ca8") : secret "image-registry-tls" not found
Sep 09 20:52:59.844702 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:52:59.844624    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d7f530ee-5bfc-4e22-acc2-328625228733-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-gqh2c\" (UID: \"d7f530ee-5bfc-4e22-acc2-328625228733\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-gqh2c"
Sep 09 20:52:59.844810 ip-10-0-138-224 kubenswrapper[2579]: E0909 20:52:59.844785    2579 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 09 20:52:59.844898 ip-10-0-138-224 kubenswrapper[2579]: E0909 20:52:59.844885    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7f530ee-5bfc-4e22-acc2-328625228733-cluster-monitoring-operator-tls podName:d7f530ee-5bfc-4e22-acc2-328625228733 nodeName:}" failed. No retries permitted until 2026-09-09 20:53:01.844863954 +0000 UTC m=+39.084958077 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/d7f530ee-5bfc-4e22-acc2-328625228733-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-gqh2c" (UID: "d7f530ee-5bfc-4e22-acc2-328625228733") : secret "cluster-monitoring-operator-tls" not found
Sep 09 20:53:00.474787 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:00.474739    2579 generic.go:356] "Generic (PLEG): container finished" podID="dfe1a979-9dff-419b-b517-93e03c19c040" containerID="b9b7894783a887cb2dbdc504d5c79c927170d8ce68e74c0b8f5e07ad21207c93" exitCode=0
Sep 09 20:53:00.475565 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:00.474836    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9rprp" event={"ID":"dfe1a979-9dff-419b-b517-93e03c19c040","Type":"ContainerDied","Data":"b9b7894783a887cb2dbdc504d5c79c927170d8ce68e74c0b8f5e07ad21207c93"}
Sep 09 20:53:00.476664 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:00.476634    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-bm5qc" event={"ID":"c31fd8ed-0c33-446d-9b0e-76fdd5935030","Type":"ContainerStarted","Data":"d86846e08bf2825932111b817805c301647dab4b9d46c6068d15829633f91f30"}
Sep 09 20:53:00.519693 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:00.519629    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-bm5qc" podStartSLOduration=2.597311318 podStartE2EDuration="3.519607778s" totalImagesPullingTime="922.29646ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:52:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:53:00.517889015 +0000 UTC m=+37.757983133" watchObservedRunningTime="2026-09-09 20:53:00.519607778 +0000 UTC m=+37.759701890"
Sep 09 20:53:01.357895 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:01.357708    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8de0dbc6-534d-4a25-b218-8e13816e7932-metrics-tls\") pod \"dns-default-zwg8k\" (UID: \"8de0dbc6-534d-4a25-b218-8e13816e7932\") " pod="openshift-dns/dns-default-zwg8k"
Sep 09 20:53:01.358053 ip-10-0-138-224 kubenswrapper[2579]: E0909 20:53:01.357858    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 20:53:01.358053 ip-10-0-138-224 kubenswrapper[2579]: E0909 20:53:01.358014    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8de0dbc6-534d-4a25-b218-8e13816e7932-metrics-tls podName:8de0dbc6-534d-4a25-b218-8e13816e7932 nodeName:}" failed. No retries permitted until 2026-09-09 20:53:05.357998286 +0000 UTC m=+42.598092383 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8de0dbc6-534d-4a25-b218-8e13816e7932-metrics-tls") pod "dns-default-zwg8k" (UID: "8de0dbc6-534d-4a25-b218-8e13816e7932") : secret "dns-default-metrics-tls" not found
Sep 09 20:53:01.480711 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:01.480680    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-h4v54" event={"ID":"5555e03e-a908-4628-a153-a8e0f4bf3f42","Type":"ContainerStarted","Data":"0b1cf9b27939c9341b9a600db03cbe0a9ff699cedd2a00dee61a82160d5f7740"}
Sep 09 20:53:01.484356 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:01.484336    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9rprp" event={"ID":"dfe1a979-9dff-419b-b517-93e03c19c040","Type":"ContainerStarted","Data":"f964c2b062cde7569d42dee1aa9f8f525d7baf2e9df6ec9b640746b6750bca3e"}
Sep 09 20:53:01.499535 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:01.499479    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-h4v54" podStartSLOduration=2.160473026 podStartE2EDuration="4.499460175s" totalImagesPullingTime="2.338987149s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:52:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:53:01.498288184 +0000 UTC m=+38.738382302" watchObservedRunningTime="2026-09-09 20:53:01.499460175 +0000 UTC m=+38.739554288"
Sep 09 20:53:01.521221 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:01.521171    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-9rprp" podStartSLOduration=7.446064968 podStartE2EDuration="38.521155382s" totalImagesPullingTime="31.075090414s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:52:23 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:53:01.51962083 +0000 UTC m=+38.759714947" watchObservedRunningTime="2026-09-09 20:53:01.521155382 +0000 UTC m=+38.761249513"
Sep 09 20:53:01.761411 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:01.761324    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/729f090c-e6f4-4757-98ec-9eba867c9ca8-registry-tls\") pod \"image-registry-64bb684d65-d9qxp\" (UID: \"729f090c-e6f4-4757-98ec-9eba867c9ca8\") " pod="openshift-image-registry/image-registry-64bb684d65-d9qxp"
Sep 09 20:53:01.761568 ip-10-0-138-224 kubenswrapper[2579]: E0909 20:53:01.761498    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 09 20:53:01.761568 ip-10-0-138-224 kubenswrapper[2579]: E0909 20:53:01.761515    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-64bb684d65-d9qxp: secret "image-registry-tls" not found
Sep 09 20:53:01.761635 ip-10-0-138-224 kubenswrapper[2579]: E0909 20:53:01.761572    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/729f090c-e6f4-4757-98ec-9eba867c9ca8-registry-tls podName:729f090c-e6f4-4757-98ec-9eba867c9ca8 nodeName:}" failed. No retries permitted until 2026-09-09 20:53:05.761556952 +0000 UTC m=+43.001651052 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/729f090c-e6f4-4757-98ec-9eba867c9ca8-registry-tls") pod "image-registry-64bb684d65-d9qxp" (UID: "729f090c-e6f4-4757-98ec-9eba867c9ca8") : secret "image-registry-tls" not found
Sep 09 20:53:01.862188 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:01.862153    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d7f530ee-5bfc-4e22-acc2-328625228733-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-gqh2c\" (UID: \"d7f530ee-5bfc-4e22-acc2-328625228733\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-gqh2c"
Sep 09 20:53:01.862355 ip-10-0-138-224 kubenswrapper[2579]: E0909 20:53:01.862332    2579 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 09 20:53:01.862473 ip-10-0-138-224 kubenswrapper[2579]: E0909 20:53:01.862411    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7f530ee-5bfc-4e22-acc2-328625228733-cluster-monitoring-operator-tls podName:d7f530ee-5bfc-4e22-acc2-328625228733 nodeName:}" failed. No retries permitted until 2026-09-09 20:53:05.862392589 +0000 UTC m=+43.102486695 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/d7f530ee-5bfc-4e22-acc2-328625228733-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-gqh2c" (UID: "d7f530ee-5bfc-4e22-acc2-328625228733") : secret "cluster-monitoring-operator-tls" not found
Sep 09 20:53:03.269965 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:03.269931    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-9z7nf"]
Sep 09 20:53:03.294140 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:03.294114    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-9z7nf"
Sep 09 20:53:03.297193 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:03.297170    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 09 20:53:03.297324 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:03.297169    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 09 20:53:03.297324 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:03.297199    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-s99rg\""
Sep 09 20:53:03.297527 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:03.297505    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 09 20:53:03.297632 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:03.297512    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 09 20:53:03.297915 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:03.297900    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-9z7nf"]
Sep 09 20:53:03.372735 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:03.372700    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8cb68ecc-5af7-4e01-8032-d41b652555ad-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9z7nf\" (UID: \"8cb68ecc-5af7-4e01-8032-d41b652555ad\") " pod="openshift-insights/insights-runtime-extractor-9z7nf"
Sep 09 20:53:03.372904 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:03.372753    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2nbn2\" (UniqueName: \"kubernetes.io/projected/8cb68ecc-5af7-4e01-8032-d41b652555ad-kube-api-access-2nbn2\") pod \"insights-runtime-extractor-9z7nf\" (UID: \"8cb68ecc-5af7-4e01-8032-d41b652555ad\") " pod="openshift-insights/insights-runtime-extractor-9z7nf"
Sep 09 20:53:03.372904 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:03.372777    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/8cb68ecc-5af7-4e01-8032-d41b652555ad-data-volume\") pod \"insights-runtime-extractor-9z7nf\" (UID: \"8cb68ecc-5af7-4e01-8032-d41b652555ad\") " pod="openshift-insights/insights-runtime-extractor-9z7nf"
Sep 09 20:53:03.372904 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:03.372888    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8cb68ecc-5af7-4e01-8032-d41b652555ad-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9z7nf\" (UID: \"8cb68ecc-5af7-4e01-8032-d41b652555ad\") " pod="openshift-insights/insights-runtime-extractor-9z7nf"
Sep 09 20:53:03.373019 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:03.372922    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/8cb68ecc-5af7-4e01-8032-d41b652555ad-crio-socket\") pod \"insights-runtime-extractor-9z7nf\" (UID: \"8cb68ecc-5af7-4e01-8032-d41b652555ad\") " pod="openshift-insights/insights-runtime-extractor-9z7nf"
Sep 09 20:53:03.473403 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:03.473370    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8cb68ecc-5af7-4e01-8032-d41b652555ad-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9z7nf\" (UID: \"8cb68ecc-5af7-4e01-8032-d41b652555ad\") " pod="openshift-insights/insights-runtime-extractor-9z7nf"
Sep 09 20:53:03.473572 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:03.473420    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2nbn2\" (UniqueName: \"kubernetes.io/projected/8cb68ecc-5af7-4e01-8032-d41b652555ad-kube-api-access-2nbn2\") pod \"insights-runtime-extractor-9z7nf\" (UID: \"8cb68ecc-5af7-4e01-8032-d41b652555ad\") " pod="openshift-insights/insights-runtime-extractor-9z7nf"
Sep 09 20:53:03.473572 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:03.473453    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/8cb68ecc-5af7-4e01-8032-d41b652555ad-data-volume\") pod \"insights-runtime-extractor-9z7nf\" (UID: \"8cb68ecc-5af7-4e01-8032-d41b652555ad\") " pod="openshift-insights/insights-runtime-extractor-9z7nf"
Sep 09 20:53:03.473572 ip-10-0-138-224 kubenswrapper[2579]: E0909 20:53:03.473536    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 20:53:03.473722 ip-10-0-138-224 kubenswrapper[2579]: E0909 20:53:03.473601    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8cb68ecc-5af7-4e01-8032-d41b652555ad-insights-runtime-extractor-tls podName:8cb68ecc-5af7-4e01-8032-d41b652555ad nodeName:}" failed. No retries permitted until 2026-09-09 20:53:03.973585353 +0000 UTC m=+41.213679453 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8cb68ecc-5af7-4e01-8032-d41b652555ad-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9z7nf" (UID: "8cb68ecc-5af7-4e01-8032-d41b652555ad") : secret "insights-runtime-extractor-tls" not found
Sep 09 20:53:03.473722 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:03.473658    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8cb68ecc-5af7-4e01-8032-d41b652555ad-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9z7nf\" (UID: \"8cb68ecc-5af7-4e01-8032-d41b652555ad\") " pod="openshift-insights/insights-runtime-extractor-9z7nf"
Sep 09 20:53:03.473722 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:03.473701    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/8cb68ecc-5af7-4e01-8032-d41b652555ad-crio-socket\") pod \"insights-runtime-extractor-9z7nf\" (UID: \"8cb68ecc-5af7-4e01-8032-d41b652555ad\") " pod="openshift-insights/insights-runtime-extractor-9z7nf"
Sep 09 20:53:03.473869 ip-10-0-138-224 kubenswrapper[2579]: E0909 20:53:03.473762    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 20:53:03.473869 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:03.473767    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/8cb68ecc-5af7-4e01-8032-d41b652555ad-data-volume\") pod \"insights-runtime-extractor-9z7nf\" (UID: \"8cb68ecc-5af7-4e01-8032-d41b652555ad\") " pod="openshift-insights/insights-runtime-extractor-9z7nf"
Sep 09 20:53:03.473869 ip-10-0-138-224 kubenswrapper[2579]: E0909 20:53:03.473820    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8cb68ecc-5af7-4e01-8032-d41b652555ad-kube-rbac-proxy-cm podName:8cb68ecc-5af7-4e01-8032-d41b652555ad nodeName:}" failed. No retries permitted until 2026-09-09 20:53:03.973805703 +0000 UTC m=+41.213899799 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/8cb68ecc-5af7-4e01-8032-d41b652555ad-kube-rbac-proxy-cm") pod "insights-runtime-extractor-9z7nf" (UID: "8cb68ecc-5af7-4e01-8032-d41b652555ad") : configmap "kube-rbac-proxy" not found
Sep 09 20:53:03.473966 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:03.473902    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/8cb68ecc-5af7-4e01-8032-d41b652555ad-crio-socket\") pod \"insights-runtime-extractor-9z7nf\" (UID: \"8cb68ecc-5af7-4e01-8032-d41b652555ad\") " pod="openshift-insights/insights-runtime-extractor-9z7nf"
Sep 09 20:53:03.489749 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:03.489721    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2nbn2\" (UniqueName: \"kubernetes.io/projected/8cb68ecc-5af7-4e01-8032-d41b652555ad-kube-api-access-2nbn2\") pod \"insights-runtime-extractor-9z7nf\" (UID: \"8cb68ecc-5af7-4e01-8032-d41b652555ad\") " pod="openshift-insights/insights-runtime-extractor-9z7nf"
Sep 09 20:53:03.792684 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:03.792649    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-fdbb8fdff-4kjtk"]
Sep 09 20:53:03.825162 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:03.825137    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-fdbb8fdff-4kjtk"]
Sep 09 20:53:03.825297 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:03.825241    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-4kjtk"
Sep 09 20:53:03.828347 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:03.828326    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Sep 09 20:53:03.828431 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:03.828358    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Sep 09 20:53:03.828635 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:03.828621    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-mc2pt\""
Sep 09 20:53:03.977233 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:03.977198    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8cb68ecc-5af7-4e01-8032-d41b652555ad-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9z7nf\" (UID: \"8cb68ecc-5af7-4e01-8032-d41b652555ad\") " pod="openshift-insights/insights-runtime-extractor-9z7nf"
Sep 09 20:53:03.977233 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:03.977234    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfn6q\" (UniqueName: \"kubernetes.io/projected/fd3e0c14-9e17-494f-b464-fd6b15e1f738-kube-api-access-jfn6q\") pod \"migrator-fdbb8fdff-4kjtk\" (UID: \"fd3e0c14-9e17-494f-b464-fd6b15e1f738\") " pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-4kjtk"
Sep 09 20:53:03.977419 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:03.977281    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8cb68ecc-5af7-4e01-8032-d41b652555ad-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9z7nf\" (UID: \"8cb68ecc-5af7-4e01-8032-d41b652555ad\") " pod="openshift-insights/insights-runtime-extractor-9z7nf"
Sep 09 20:53:03.977419 ip-10-0-138-224 kubenswrapper[2579]: E0909 20:53:03.977320    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 20:53:03.977419 ip-10-0-138-224 kubenswrapper[2579]: E0909 20:53:03.977374    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8cb68ecc-5af7-4e01-8032-d41b652555ad-kube-rbac-proxy-cm podName:8cb68ecc-5af7-4e01-8032-d41b652555ad nodeName:}" failed. No retries permitted until 2026-09-09 20:53:04.977360318 +0000 UTC m=+42.217454442 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/8cb68ecc-5af7-4e01-8032-d41b652555ad-kube-rbac-proxy-cm") pod "insights-runtime-extractor-9z7nf" (UID: "8cb68ecc-5af7-4e01-8032-d41b652555ad") : configmap "kube-rbac-proxy" not found
Sep 09 20:53:03.977419 ip-10-0-138-224 kubenswrapper[2579]: E0909 20:53:03.977386    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 20:53:03.977606 ip-10-0-138-224 kubenswrapper[2579]: E0909 20:53:03.977425    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8cb68ecc-5af7-4e01-8032-d41b652555ad-insights-runtime-extractor-tls podName:8cb68ecc-5af7-4e01-8032-d41b652555ad nodeName:}" failed. No retries permitted until 2026-09-09 20:53:04.97741326 +0000 UTC m=+42.217507356 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8cb68ecc-5af7-4e01-8032-d41b652555ad-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9z7nf" (UID: "8cb68ecc-5af7-4e01-8032-d41b652555ad") : secret "insights-runtime-extractor-tls" not found
Sep 09 20:53:04.078431 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:04.078358    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfn6q\" (UniqueName: \"kubernetes.io/projected/fd3e0c14-9e17-494f-b464-fd6b15e1f738-kube-api-access-jfn6q\") pod \"migrator-fdbb8fdff-4kjtk\" (UID: \"fd3e0c14-9e17-494f-b464-fd6b15e1f738\") " pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-4kjtk"
Sep 09 20:53:04.088316 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:04.088285    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfn6q\" (UniqueName: \"kubernetes.io/projected/fd3e0c14-9e17-494f-b464-fd6b15e1f738-kube-api-access-jfn6q\") pod \"migrator-fdbb8fdff-4kjtk\" (UID: \"fd3e0c14-9e17-494f-b464-fd6b15e1f738\") " pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-4kjtk"
Sep 09 20:53:04.134669 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:04.134646    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-4kjtk"
Sep 09 20:53:04.456969 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:04.456929    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-fdbb8fdff-4kjtk"]
Sep 09 20:53:04.460196 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:53:04.460165    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd3e0c14_9e17_494f_b464_fd6b15e1f738.slice/crio-d2c4d935c4990c3dde4b5392aba846fa02738e05b27bb15dfca79fc6c6b2cd64 WatchSource:0}: Error finding container d2c4d935c4990c3dde4b5392aba846fa02738e05b27bb15dfca79fc6c6b2cd64: Status 404 returned error can't find the container with id d2c4d935c4990c3dde4b5392aba846fa02738e05b27bb15dfca79fc6c6b2cd64
Sep 09 20:53:04.492087 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:04.492062    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-4kjtk" event={"ID":"fd3e0c14-9e17-494f-b464-fd6b15e1f738","Type":"ContainerStarted","Data":"d2c4d935c4990c3dde4b5392aba846fa02738e05b27bb15dfca79fc6c6b2cd64"}
Sep 09 20:53:04.987286 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:04.987248    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8cb68ecc-5af7-4e01-8032-d41b652555ad-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9z7nf\" (UID: \"8cb68ecc-5af7-4e01-8032-d41b652555ad\") " pod="openshift-insights/insights-runtime-extractor-9z7nf"
Sep 09 20:53:04.987461 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:04.987316    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8cb68ecc-5af7-4e01-8032-d41b652555ad-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9z7nf\" (UID: \"8cb68ecc-5af7-4e01-8032-d41b652555ad\") " pod="openshift-insights/insights-runtime-extractor-9z7nf"
Sep 09 20:53:04.987461 ip-10-0-138-224 kubenswrapper[2579]: E0909 20:53:04.987386    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 20:53:04.987461 ip-10-0-138-224 kubenswrapper[2579]: E0909 20:53:04.987421    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 20:53:04.987599 ip-10-0-138-224 kubenswrapper[2579]: E0909 20:53:04.987483    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8cb68ecc-5af7-4e01-8032-d41b652555ad-kube-rbac-proxy-cm podName:8cb68ecc-5af7-4e01-8032-d41b652555ad nodeName:}" failed. No retries permitted until 2026-09-09 20:53:06.987467357 +0000 UTC m=+44.227561457 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/8cb68ecc-5af7-4e01-8032-d41b652555ad-kube-rbac-proxy-cm") pod "insights-runtime-extractor-9z7nf" (UID: "8cb68ecc-5af7-4e01-8032-d41b652555ad") : configmap "kube-rbac-proxy" not found
Sep 09 20:53:04.987599 ip-10-0-138-224 kubenswrapper[2579]: E0909 20:53:04.987499    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8cb68ecc-5af7-4e01-8032-d41b652555ad-insights-runtime-extractor-tls podName:8cb68ecc-5af7-4e01-8032-d41b652555ad nodeName:}" failed. No retries permitted until 2026-09-09 20:53:06.987493449 +0000 UTC m=+44.227587544 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8cb68ecc-5af7-4e01-8032-d41b652555ad-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9z7nf" (UID: "8cb68ecc-5af7-4e01-8032-d41b652555ad") : secret "insights-runtime-extractor-tls" not found
Sep 09 20:53:05.046555 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:05.046532    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-88pfs_ddb26243-140d-4ac9-8a3d-4079559bd0e7/dns-node-resolver/0.log"
Sep 09 20:53:05.390725 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:05.390638    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8de0dbc6-534d-4a25-b218-8e13816e7932-metrics-tls\") pod \"dns-default-zwg8k\" (UID: \"8de0dbc6-534d-4a25-b218-8e13816e7932\") " pod="openshift-dns/dns-default-zwg8k"
Sep 09 20:53:05.390862 ip-10-0-138-224 kubenswrapper[2579]: E0909 20:53:05.390789    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 20:53:05.390862 ip-10-0-138-224 kubenswrapper[2579]: E0909 20:53:05.390853    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8de0dbc6-534d-4a25-b218-8e13816e7932-metrics-tls podName:8de0dbc6-534d-4a25-b218-8e13816e7932 nodeName:}" failed. No retries permitted until 2026-09-09 20:53:13.390837622 +0000 UTC m=+50.630931721 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8de0dbc6-534d-4a25-b218-8e13816e7932-metrics-tls") pod "dns-default-zwg8k" (UID: "8de0dbc6-534d-4a25-b218-8e13816e7932") : secret "dns-default-metrics-tls" not found
Sep 09 20:53:05.793588 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:05.793538    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/729f090c-e6f4-4757-98ec-9eba867c9ca8-registry-tls\") pod \"image-registry-64bb684d65-d9qxp\" (UID: \"729f090c-e6f4-4757-98ec-9eba867c9ca8\") " pod="openshift-image-registry/image-registry-64bb684d65-d9qxp"
Sep 09 20:53:05.793934 ip-10-0-138-224 kubenswrapper[2579]: E0909 20:53:05.793709    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 09 20:53:05.793934 ip-10-0-138-224 kubenswrapper[2579]: E0909 20:53:05.793724    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-64bb684d65-d9qxp: secret "image-registry-tls" not found
Sep 09 20:53:05.793934 ip-10-0-138-224 kubenswrapper[2579]: E0909 20:53:05.793791    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/729f090c-e6f4-4757-98ec-9eba867c9ca8-registry-tls podName:729f090c-e6f4-4757-98ec-9eba867c9ca8 nodeName:}" failed. No retries permitted until 2026-09-09 20:53:13.793774119 +0000 UTC m=+51.033868218 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/729f090c-e6f4-4757-98ec-9eba867c9ca8-registry-tls") pod "image-registry-64bb684d65-d9qxp" (UID: "729f090c-e6f4-4757-98ec-9eba867c9ca8") : secret "image-registry-tls" not found
Sep 09 20:53:05.894852 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:05.894823    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d7f530ee-5bfc-4e22-acc2-328625228733-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-gqh2c\" (UID: \"d7f530ee-5bfc-4e22-acc2-328625228733\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-gqh2c"
Sep 09 20:53:05.894950 ip-10-0-138-224 kubenswrapper[2579]: E0909 20:53:05.894931    2579 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 09 20:53:05.894994 ip-10-0-138-224 kubenswrapper[2579]: E0909 20:53:05.894979    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7f530ee-5bfc-4e22-acc2-328625228733-cluster-monitoring-operator-tls podName:d7f530ee-5bfc-4e22-acc2-328625228733 nodeName:}" failed. No retries permitted until 2026-09-09 20:53:13.89496754 +0000 UTC m=+51.135061637 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/d7f530ee-5bfc-4e22-acc2-328625228733-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-gqh2c" (UID: "d7f530ee-5bfc-4e22-acc2-328625228733") : secret "cluster-monitoring-operator-tls" not found
Sep 09 20:53:06.234234 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:06.234215    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-tml69_169cade8-7395-4b62-9544-ff0c47022a89/node-ca/0.log"
Sep 09 20:53:06.499231 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:06.499153    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-4kjtk" event={"ID":"fd3e0c14-9e17-494f-b464-fd6b15e1f738","Type":"ContainerStarted","Data":"a4361d88797094a2ba091eb3ba576d39b51649399fc591264f59d9f7a6a3c695"}
Sep 09 20:53:06.499231 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:06.499201    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-4kjtk" event={"ID":"fd3e0c14-9e17-494f-b464-fd6b15e1f738","Type":"ContainerStarted","Data":"01536b89bee6816bb60eb9ab12baa576ca43142ba2c9fc881698579900efb8db"}
Sep 09 20:53:07.002104 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:07.002079    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8cb68ecc-5af7-4e01-8032-d41b652555ad-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9z7nf\" (UID: \"8cb68ecc-5af7-4e01-8032-d41b652555ad\") " pod="openshift-insights/insights-runtime-extractor-9z7nf"
Sep 09 20:53:07.002461 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:07.002138    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8cb68ecc-5af7-4e01-8032-d41b652555ad-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9z7nf\" (UID: \"8cb68ecc-5af7-4e01-8032-d41b652555ad\") " pod="openshift-insights/insights-runtime-extractor-9z7nf"
Sep 09 20:53:07.002461 ip-10-0-138-224 kubenswrapper[2579]: E0909 20:53:07.002205    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 20:53:07.002461 ip-10-0-138-224 kubenswrapper[2579]: E0909 20:53:07.002231    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 20:53:07.002461 ip-10-0-138-224 kubenswrapper[2579]: E0909 20:53:07.002271    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8cb68ecc-5af7-4e01-8032-d41b652555ad-kube-rbac-proxy-cm podName:8cb68ecc-5af7-4e01-8032-d41b652555ad nodeName:}" failed. No retries permitted until 2026-09-09 20:53:11.002256112 +0000 UTC m=+48.242350212 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/8cb68ecc-5af7-4e01-8032-d41b652555ad-kube-rbac-proxy-cm") pod "insights-runtime-extractor-9z7nf" (UID: "8cb68ecc-5af7-4e01-8032-d41b652555ad") : configmap "kube-rbac-proxy" not found
Sep 09 20:53:07.002461 ip-10-0-138-224 kubenswrapper[2579]: E0909 20:53:07.002285    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8cb68ecc-5af7-4e01-8032-d41b652555ad-insights-runtime-extractor-tls podName:8cb68ecc-5af7-4e01-8032-d41b652555ad nodeName:}" failed. No retries permitted until 2026-09-09 20:53:11.002278914 +0000 UTC m=+48.242373009 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8cb68ecc-5af7-4e01-8032-d41b652555ad-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9z7nf" (UID: "8cb68ecc-5af7-4e01-8032-d41b652555ad") : secret "insights-runtime-extractor-tls" not found
Sep 09 20:53:07.031124 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:07.031098    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-h4v54_5555e03e-a908-4628-a153-a8e0f4bf3f42/serve-healthcheck-canary/0.log"
Sep 09 20:53:11.035063 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:11.035028    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8cb68ecc-5af7-4e01-8032-d41b652555ad-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9z7nf\" (UID: \"8cb68ecc-5af7-4e01-8032-d41b652555ad\") " pod="openshift-insights/insights-runtime-extractor-9z7nf"
Sep 09 20:53:11.035489 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:11.035108    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8cb68ecc-5af7-4e01-8032-d41b652555ad-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9z7nf\" (UID: \"8cb68ecc-5af7-4e01-8032-d41b652555ad\") " pod="openshift-insights/insights-runtime-extractor-9z7nf"
Sep 09 20:53:11.035489 ip-10-0-138-224 kubenswrapper[2579]: E0909 20:53:11.035177    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 20:53:11.035489 ip-10-0-138-224 kubenswrapper[2579]: E0909 20:53:11.035197    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 20:53:11.035489 ip-10-0-138-224 kubenswrapper[2579]: E0909 20:53:11.035242    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8cb68ecc-5af7-4e01-8032-d41b652555ad-insights-runtime-extractor-tls podName:8cb68ecc-5af7-4e01-8032-d41b652555ad nodeName:}" failed. No retries permitted until 2026-09-09 20:53:19.035226925 +0000 UTC m=+56.275321020 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8cb68ecc-5af7-4e01-8032-d41b652555ad-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9z7nf" (UID: "8cb68ecc-5af7-4e01-8032-d41b652555ad") : secret "insights-runtime-extractor-tls" not found
Sep 09 20:53:11.035489 ip-10-0-138-224 kubenswrapper[2579]: E0909 20:53:11.035257    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8cb68ecc-5af7-4e01-8032-d41b652555ad-kube-rbac-proxy-cm podName:8cb68ecc-5af7-4e01-8032-d41b652555ad nodeName:}" failed. No retries permitted until 2026-09-09 20:53:19.035249591 +0000 UTC m=+56.275343687 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/8cb68ecc-5af7-4e01-8032-d41b652555ad-kube-rbac-proxy-cm") pod "insights-runtime-extractor-9z7nf" (UID: "8cb68ecc-5af7-4e01-8032-d41b652555ad") : configmap "kube-rbac-proxy" not found
Sep 09 20:53:13.453101 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:13.453067    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8de0dbc6-534d-4a25-b218-8e13816e7932-metrics-tls\") pod \"dns-default-zwg8k\" (UID: \"8de0dbc6-534d-4a25-b218-8e13816e7932\") " pod="openshift-dns/dns-default-zwg8k"
Sep 09 20:53:13.455490 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:13.455460    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8de0dbc6-534d-4a25-b218-8e13816e7932-metrics-tls\") pod \"dns-default-zwg8k\" (UID: \"8de0dbc6-534d-4a25-b218-8e13816e7932\") " pod="openshift-dns/dns-default-zwg8k"
Sep 09 20:53:13.507209 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:13.507177    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-zwg8k"
Sep 09 20:53:13.789039 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:13.788947    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-4kjtk" podStartSLOduration=9.092939073 podStartE2EDuration="10.788929449s" totalImagesPullingTime="1.695990376s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:53:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:53:06.52228584 +0000 UTC m=+43.762379961" watchObservedRunningTime="2026-09-09 20:53:13.788929449 +0000 UTC m=+51.029023567"
Sep 09 20:53:13.789266 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:13.789252    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-zwg8k"]
Sep 09 20:53:13.792240 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:53:13.792215    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8de0dbc6_534d_4a25_b218_8e13816e7932.slice/crio-120535b63ba915b64723603847a721e530511922efdc3f2f6f9a915fae40e933 WatchSource:0}: Error finding container 120535b63ba915b64723603847a721e530511922efdc3f2f6f9a915fae40e933: Status 404 returned error can't find the container with id 120535b63ba915b64723603847a721e530511922efdc3f2f6f9a915fae40e933
Sep 09 20:53:13.857080 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:13.857051    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/729f090c-e6f4-4757-98ec-9eba867c9ca8-registry-tls\") pod \"image-registry-64bb684d65-d9qxp\" (UID: \"729f090c-e6f4-4757-98ec-9eba867c9ca8\") " pod="openshift-image-registry/image-registry-64bb684d65-d9qxp"
Sep 09 20:53:13.859490 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:13.859468    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/729f090c-e6f4-4757-98ec-9eba867c9ca8-registry-tls\") pod \"image-registry-64bb684d65-d9qxp\" (UID: \"729f090c-e6f4-4757-98ec-9eba867c9ca8\") " pod="openshift-image-registry/image-registry-64bb684d65-d9qxp"
Sep 09 20:53:13.928081 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:13.928053    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-64bb684d65-d9qxp"
Sep 09 20:53:13.957507 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:13.957483    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d7f530ee-5bfc-4e22-acc2-328625228733-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-gqh2c\" (UID: \"d7f530ee-5bfc-4e22-acc2-328625228733\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-gqh2c"
Sep 09 20:53:13.957636 ip-10-0-138-224 kubenswrapper[2579]: E0909 20:53:13.957617    2579 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 09 20:53:13.957686 ip-10-0-138-224 kubenswrapper[2579]: E0909 20:53:13.957681    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7f530ee-5bfc-4e22-acc2-328625228733-cluster-monitoring-operator-tls podName:d7f530ee-5bfc-4e22-acc2-328625228733 nodeName:}" failed. No retries permitted until 2026-09-09 20:53:29.957665249 +0000 UTC m=+67.197759349 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/d7f530ee-5bfc-4e22-acc2-328625228733-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-gqh2c" (UID: "d7f530ee-5bfc-4e22-acc2-328625228733") : secret "cluster-monitoring-operator-tls" not found
Sep 09 20:53:14.215721 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:14.215696    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-64bb684d65-d9qxp"]
Sep 09 20:53:14.520772 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:14.520674    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-64bb684d65-d9qxp" event={"ID":"729f090c-e6f4-4757-98ec-9eba867c9ca8","Type":"ContainerStarted","Data":"3e1bbc2bdd6373de148fbea32c88b0e090045f1730a0dfd2ead524a687abb8d1"}
Sep 09 20:53:14.521186 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:14.520777    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-64bb684d65-d9qxp" event={"ID":"729f090c-e6f4-4757-98ec-9eba867c9ca8","Type":"ContainerStarted","Data":"88d8ade4ba59262d8b35b8eca4aed1e85643c86efce04c2b89e3d4558f155576"}
Sep 09 20:53:14.521186 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:14.520807    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-64bb684d65-d9qxp"
Sep 09 20:53:14.522472 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:14.522395    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-zwg8k" event={"ID":"8de0dbc6-534d-4a25-b218-8e13816e7932","Type":"ContainerStarted","Data":"120535b63ba915b64723603847a721e530511922efdc3f2f6f9a915fae40e933"}
Sep 09 20:53:14.546681 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:14.546628    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-64bb684d65-d9qxp" podStartSLOduration=17.546614801 podStartE2EDuration="17.546614801s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:52:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:53:14.54478873 +0000 UTC m=+51.784882850" watchObservedRunningTime="2026-09-09 20:53:14.546614801 +0000 UTC m=+51.786708918"
Sep 09 20:53:16.530925 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:16.530883    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-zwg8k" event={"ID":"8de0dbc6-534d-4a25-b218-8e13816e7932","Type":"ContainerStarted","Data":"cac7a3c01f2856138d6eaa0b0f15d4c7e343447d3013957c45e2096edd318fc6"}
Sep 09 20:53:16.531264 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:16.530944    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-zwg8k" event={"ID":"8de0dbc6-534d-4a25-b218-8e13816e7932","Type":"ContainerStarted","Data":"40e8e9e736f43170148cbd965659b30c35c0a1d894b6689585cf377a3d5c44f6"}
Sep 09 20:53:16.531264 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:16.530990    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-zwg8k"
Sep 09 20:53:16.551974 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:16.551928    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-zwg8k" podStartSLOduration=17.456316055 podStartE2EDuration="19.551915448s" totalImagesPullingTime="2.095599393s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:52:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:53:16.550853803 +0000 UTC m=+53.790947921" watchObservedRunningTime="2026-09-09 20:53:16.551915448 +0000 UTC m=+53.792009560"
Sep 09 20:53:19.093670 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:19.093630    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8cb68ecc-5af7-4e01-8032-d41b652555ad-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9z7nf\" (UID: \"8cb68ecc-5af7-4e01-8032-d41b652555ad\") " pod="openshift-insights/insights-runtime-extractor-9z7nf"
Sep 09 20:53:19.094144 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:19.093708    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8cb68ecc-5af7-4e01-8032-d41b652555ad-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9z7nf\" (UID: \"8cb68ecc-5af7-4e01-8032-d41b652555ad\") " pod="openshift-insights/insights-runtime-extractor-9z7nf"
Sep 09 20:53:19.094144 ip-10-0-138-224 kubenswrapper[2579]: E0909 20:53:19.093852    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 20:53:19.094144 ip-10-0-138-224 kubenswrapper[2579]: E0909 20:53:19.093931    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8cb68ecc-5af7-4e01-8032-d41b652555ad-insights-runtime-extractor-tls podName:8cb68ecc-5af7-4e01-8032-d41b652555ad nodeName:}" failed. No retries permitted until 2026-09-09 20:53:35.093908637 +0000 UTC m=+72.334002750 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8cb68ecc-5af7-4e01-8032-d41b652555ad-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9z7nf" (UID: "8cb68ecc-5af7-4e01-8032-d41b652555ad") : secret "insights-runtime-extractor-tls" not found
Sep 09 20:53:19.094306 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:19.094188    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8cb68ecc-5af7-4e01-8032-d41b652555ad-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9z7nf\" (UID: \"8cb68ecc-5af7-4e01-8032-d41b652555ad\") " pod="openshift-insights/insights-runtime-extractor-9z7nf"
Sep 09 20:53:23.452668 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:23.452638    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-9dcqf"
Sep 09 20:53:26.538766 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:26.538732    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-zwg8k"
Sep 09 20:53:26.676758 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:26.676722    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7c484d6b5b-s2qds"]
Sep 09 20:53:26.681312 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:26.681289    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7c484d6b5b-s2qds"
Sep 09 20:53:26.688627 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:26.688603    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 09 20:53:26.688762 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:26.688743    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 09 20:53:26.689464 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:26.689423    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-64bb684d65-d9qxp"]
Sep 09 20:53:26.689731 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:26.689708    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 09 20:53:26.689844 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:26.689728    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-jl5pd\""
Sep 09 20:53:26.692501 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:26.692484    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Sep 09 20:53:26.706506 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:26.706484    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7c484d6b5b-s2qds"]
Sep 09 20:53:26.748073 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:26.748048    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9g2bz\" (UniqueName: \"kubernetes.io/projected/255102d9-fa66-4ec7-aa03-f8fab1119cd2-kube-api-access-9g2bz\") pod \"managed-serviceaccount-addon-agent-7c484d6b5b-s2qds\" (UID: \"255102d9-fa66-4ec7-aa03-f8fab1119cd2\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7c484d6b5b-s2qds"
Sep 09 20:53:26.748190 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:26.748095    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/255102d9-fa66-4ec7-aa03-f8fab1119cd2-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-7c484d6b5b-s2qds\" (UID: \"255102d9-fa66-4ec7-aa03-f8fab1119cd2\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7c484d6b5b-s2qds"
Sep 09 20:53:26.799804 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:26.799726    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6f4bd54dd8-fqmrm"]
Sep 09 20:53:26.802693 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:26.802671    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6f4bd54dd8-fqmrm"
Sep 09 20:53:26.822865 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:26.822834    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6f4bd54dd8-fqmrm"]
Sep 09 20:53:26.849151 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:26.849124    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0a4dc7da-cdd3-433e-8a8f-c7726607feac-ca-trust-extracted\") pod \"image-registry-6f4bd54dd8-fqmrm\" (UID: \"0a4dc7da-cdd3-433e-8a8f-c7726607feac\") " pod="openshift-image-registry/image-registry-6f4bd54dd8-fqmrm"
Sep 09 20:53:26.849245 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:26.849156    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0a4dc7da-cdd3-433e-8a8f-c7726607feac-installation-pull-secrets\") pod \"image-registry-6f4bd54dd8-fqmrm\" (UID: \"0a4dc7da-cdd3-433e-8a8f-c7726607feac\") " pod="openshift-image-registry/image-registry-6f4bd54dd8-fqmrm"
Sep 09 20:53:26.849245 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:26.849183    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0a4dc7da-cdd3-433e-8a8f-c7726607feac-registry-certificates\") pod \"image-registry-6f4bd54dd8-fqmrm\" (UID: \"0a4dc7da-cdd3-433e-8a8f-c7726607feac\") " pod="openshift-image-registry/image-registry-6f4bd54dd8-fqmrm"
Sep 09 20:53:26.849351 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:26.849285    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0a4dc7da-cdd3-433e-8a8f-c7726607feac-image-registry-private-configuration\") pod \"image-registry-6f4bd54dd8-fqmrm\" (UID: \"0a4dc7da-cdd3-433e-8a8f-c7726607feac\") " pod="openshift-image-registry/image-registry-6f4bd54dd8-fqmrm"
Sep 09 20:53:26.849351 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:26.849308    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9g2bz\" (UniqueName: \"kubernetes.io/projected/255102d9-fa66-4ec7-aa03-f8fab1119cd2-kube-api-access-9g2bz\") pod \"managed-serviceaccount-addon-agent-7c484d6b5b-s2qds\" (UID: \"255102d9-fa66-4ec7-aa03-f8fab1119cd2\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7c484d6b5b-s2qds"
Sep 09 20:53:26.849351 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:26.849325    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0a4dc7da-cdd3-433e-8a8f-c7726607feac-trusted-ca\") pod \"image-registry-6f4bd54dd8-fqmrm\" (UID: \"0a4dc7da-cdd3-433e-8a8f-c7726607feac\") " pod="openshift-image-registry/image-registry-6f4bd54dd8-fqmrm"
Sep 09 20:53:26.849351 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:26.849342    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/255102d9-fa66-4ec7-aa03-f8fab1119cd2-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-7c484d6b5b-s2qds\" (UID: \"255102d9-fa66-4ec7-aa03-f8fab1119cd2\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7c484d6b5b-s2qds"
Sep 09 20:53:26.849523 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:26.849362    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0a4dc7da-cdd3-433e-8a8f-c7726607feac-registry-tls\") pod \"image-registry-6f4bd54dd8-fqmrm\" (UID: \"0a4dc7da-cdd3-433e-8a8f-c7726607feac\") " pod="openshift-image-registry/image-registry-6f4bd54dd8-fqmrm"
Sep 09 20:53:26.849523 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:26.849387    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lf2tr\" (UniqueName: \"kubernetes.io/projected/0a4dc7da-cdd3-433e-8a8f-c7726607feac-kube-api-access-lf2tr\") pod \"image-registry-6f4bd54dd8-fqmrm\" (UID: \"0a4dc7da-cdd3-433e-8a8f-c7726607feac\") " pod="openshift-image-registry/image-registry-6f4bd54dd8-fqmrm"
Sep 09 20:53:26.849523 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:26.849492    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0a4dc7da-cdd3-433e-8a8f-c7726607feac-bound-sa-token\") pod \"image-registry-6f4bd54dd8-fqmrm\" (UID: \"0a4dc7da-cdd3-433e-8a8f-c7726607feac\") " pod="openshift-image-registry/image-registry-6f4bd54dd8-fqmrm"
Sep 09 20:53:26.851906 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:26.851878    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/255102d9-fa66-4ec7-aa03-f8fab1119cd2-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-7c484d6b5b-s2qds\" (UID: \"255102d9-fa66-4ec7-aa03-f8fab1119cd2\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7c484d6b5b-s2qds"
Sep 09 20:53:26.865158 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:26.865134    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9g2bz\" (UniqueName: \"kubernetes.io/projected/255102d9-fa66-4ec7-aa03-f8fab1119cd2-kube-api-access-9g2bz\") pod \"managed-serviceaccount-addon-agent-7c484d6b5b-s2qds\" (UID: \"255102d9-fa66-4ec7-aa03-f8fab1119cd2\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7c484d6b5b-s2qds"
Sep 09 20:53:26.950086 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:26.950044    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0a4dc7da-cdd3-433e-8a8f-c7726607feac-image-registry-private-configuration\") pod \"image-registry-6f4bd54dd8-fqmrm\" (UID: \"0a4dc7da-cdd3-433e-8a8f-c7726607feac\") " pod="openshift-image-registry/image-registry-6f4bd54dd8-fqmrm"
Sep 09 20:53:26.950086 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:26.950092    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0a4dc7da-cdd3-433e-8a8f-c7726607feac-trusted-ca\") pod \"image-registry-6f4bd54dd8-fqmrm\" (UID: \"0a4dc7da-cdd3-433e-8a8f-c7726607feac\") " pod="openshift-image-registry/image-registry-6f4bd54dd8-fqmrm"
Sep 09 20:53:26.950333 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:26.950121    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0a4dc7da-cdd3-433e-8a8f-c7726607feac-registry-tls\") pod \"image-registry-6f4bd54dd8-fqmrm\" (UID: \"0a4dc7da-cdd3-433e-8a8f-c7726607feac\") " pod="openshift-image-registry/image-registry-6f4bd54dd8-fqmrm"
Sep 09 20:53:26.950333 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:26.950143    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lf2tr\" (UniqueName: \"kubernetes.io/projected/0a4dc7da-cdd3-433e-8a8f-c7726607feac-kube-api-access-lf2tr\") pod \"image-registry-6f4bd54dd8-fqmrm\" (UID: \"0a4dc7da-cdd3-433e-8a8f-c7726607feac\") " pod="openshift-image-registry/image-registry-6f4bd54dd8-fqmrm"
Sep 09 20:53:26.950333 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:26.950315    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0a4dc7da-cdd3-433e-8a8f-c7726607feac-bound-sa-token\") pod \"image-registry-6f4bd54dd8-fqmrm\" (UID: \"0a4dc7da-cdd3-433e-8a8f-c7726607feac\") " pod="openshift-image-registry/image-registry-6f4bd54dd8-fqmrm"
Sep 09 20:53:26.950530 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:26.950365    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0a4dc7da-cdd3-433e-8a8f-c7726607feac-ca-trust-extracted\") pod \"image-registry-6f4bd54dd8-fqmrm\" (UID: \"0a4dc7da-cdd3-433e-8a8f-c7726607feac\") " pod="openshift-image-registry/image-registry-6f4bd54dd8-fqmrm"
Sep 09 20:53:26.950530 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:26.950409    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0a4dc7da-cdd3-433e-8a8f-c7726607feac-installation-pull-secrets\") pod \"image-registry-6f4bd54dd8-fqmrm\" (UID: \"0a4dc7da-cdd3-433e-8a8f-c7726607feac\") " pod="openshift-image-registry/image-registry-6f4bd54dd8-fqmrm"
Sep 09 20:53:26.950530 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:26.950474    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0a4dc7da-cdd3-433e-8a8f-c7726607feac-registry-certificates\") pod \"image-registry-6f4bd54dd8-fqmrm\" (UID: \"0a4dc7da-cdd3-433e-8a8f-c7726607feac\") " pod="openshift-image-registry/image-registry-6f4bd54dd8-fqmrm"
Sep 09 20:53:26.950834 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:26.950801    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0a4dc7da-cdd3-433e-8a8f-c7726607feac-ca-trust-extracted\") pod \"image-registry-6f4bd54dd8-fqmrm\" (UID: \"0a4dc7da-cdd3-433e-8a8f-c7726607feac\") " pod="openshift-image-registry/image-registry-6f4bd54dd8-fqmrm"
Sep 09 20:53:26.951458 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:26.951406    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0a4dc7da-cdd3-433e-8a8f-c7726607feac-trusted-ca\") pod \"image-registry-6f4bd54dd8-fqmrm\" (UID: \"0a4dc7da-cdd3-433e-8a8f-c7726607feac\") " pod="openshift-image-registry/image-registry-6f4bd54dd8-fqmrm"
Sep 09 20:53:26.951782 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:26.951764    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0a4dc7da-cdd3-433e-8a8f-c7726607feac-registry-certificates\") pod \"image-registry-6f4bd54dd8-fqmrm\" (UID: \"0a4dc7da-cdd3-433e-8a8f-c7726607feac\") " pod="openshift-image-registry/image-registry-6f4bd54dd8-fqmrm"
Sep 09 20:53:26.952805 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:26.952783    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0a4dc7da-cdd3-433e-8a8f-c7726607feac-registry-tls\") pod \"image-registry-6f4bd54dd8-fqmrm\" (UID: \"0a4dc7da-cdd3-433e-8a8f-c7726607feac\") " pod="openshift-image-registry/image-registry-6f4bd54dd8-fqmrm"
Sep 09 20:53:26.952891 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:26.952874    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0a4dc7da-cdd3-433e-8a8f-c7726607feac-image-registry-private-configuration\") pod \"image-registry-6f4bd54dd8-fqmrm\" (UID: \"0a4dc7da-cdd3-433e-8a8f-c7726607feac\") " pod="openshift-image-registry/image-registry-6f4bd54dd8-fqmrm"
Sep 09 20:53:26.952942 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:26.952894    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0a4dc7da-cdd3-433e-8a8f-c7726607feac-installation-pull-secrets\") pod \"image-registry-6f4bd54dd8-fqmrm\" (UID: \"0a4dc7da-cdd3-433e-8a8f-c7726607feac\") " pod="openshift-image-registry/image-registry-6f4bd54dd8-fqmrm"
Sep 09 20:53:26.960109 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:26.960082    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lf2tr\" (UniqueName: \"kubernetes.io/projected/0a4dc7da-cdd3-433e-8a8f-c7726607feac-kube-api-access-lf2tr\") pod \"image-registry-6f4bd54dd8-fqmrm\" (UID: \"0a4dc7da-cdd3-433e-8a8f-c7726607feac\") " pod="openshift-image-registry/image-registry-6f4bd54dd8-fqmrm"
Sep 09 20:53:26.960557 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:26.960538    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0a4dc7da-cdd3-433e-8a8f-c7726607feac-bound-sa-token\") pod \"image-registry-6f4bd54dd8-fqmrm\" (UID: \"0a4dc7da-cdd3-433e-8a8f-c7726607feac\") " pod="openshift-image-registry/image-registry-6f4bd54dd8-fqmrm"
Sep 09 20:53:26.999789 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:26.999763    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7c484d6b5b-s2qds"
Sep 09 20:53:27.111266 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:27.111178    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6f4bd54dd8-fqmrm"
Sep 09 20:53:27.271085 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:27.271053    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7c484d6b5b-s2qds"]
Sep 09 20:53:27.273843 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:53:27.273813    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod255102d9_fa66_4ec7_aa03_f8fab1119cd2.slice/crio-69ea32acd2646f4c47b43f1d5687b1534206c5bc9d9175cf8053be03df239479 WatchSource:0}: Error finding container 69ea32acd2646f4c47b43f1d5687b1534206c5bc9d9175cf8053be03df239479: Status 404 returned error can't find the container with id 69ea32acd2646f4c47b43f1d5687b1534206c5bc9d9175cf8053be03df239479
Sep 09 20:53:27.383321 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:27.383247    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6f4bd54dd8-fqmrm"]
Sep 09 20:53:27.386188 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:53:27.386163    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a4dc7da_cdd3_433e_8a8f_c7726607feac.slice/crio-10ca42fccf84c2499f593c16d5c29198c74183e5faa358887bd367a35e97d0a0 WatchSource:0}: Error finding container 10ca42fccf84c2499f593c16d5c29198c74183e5faa358887bd367a35e97d0a0: Status 404 returned error can't find the container with id 10ca42fccf84c2499f593c16d5c29198c74183e5faa358887bd367a35e97d0a0
Sep 09 20:53:27.568871 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:27.568839    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7c484d6b5b-s2qds" event={"ID":"255102d9-fa66-4ec7-aa03-f8fab1119cd2","Type":"ContainerStarted","Data":"69ea32acd2646f4c47b43f1d5687b1534206c5bc9d9175cf8053be03df239479"}
Sep 09 20:53:27.570476 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:27.570451    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6f4bd54dd8-fqmrm" event={"ID":"0a4dc7da-cdd3-433e-8a8f-c7726607feac","Type":"ContainerStarted","Data":"5d974748580ad6d5644225bb92b6cd83722869657bc8d5ffab2d6d4efb467cd3"}
Sep 09 20:53:27.570607 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:27.570491    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6f4bd54dd8-fqmrm" event={"ID":"0a4dc7da-cdd3-433e-8a8f-c7726607feac","Type":"ContainerStarted","Data":"10ca42fccf84c2499f593c16d5c29198c74183e5faa358887bd367a35e97d0a0"}
Sep 09 20:53:27.570607 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:27.570589    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6f4bd54dd8-fqmrm"
Sep 09 20:53:27.594471 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:27.594404    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-6f4bd54dd8-fqmrm" podStartSLOduration=1.594388927 podStartE2EDuration="1.594388927s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:53:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:53:27.592719424 +0000 UTC m=+64.832813556" watchObservedRunningTime="2026-09-09 20:53:27.594388927 +0000 UTC m=+64.834483045"
Sep 09 20:53:28.058991 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:28.058952    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c6fd107c-ea2f-4e85-9a32-1e11f0858fd3-metrics-certs\") pod \"network-metrics-daemon-ws8hc\" (UID: \"c6fd107c-ea2f-4e85-9a32-1e11f0858fd3\") " pod="openshift-multus/network-metrics-daemon-ws8hc"
Sep 09 20:53:28.059223 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:28.059023    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/732d9d71-d680-4291-aaee-028307ef79dc-original-pull-secret\") pod \"global-pull-secret-syncer-gz9zx\" (UID: \"732d9d71-d680-4291-aaee-028307ef79dc\") " pod="kube-system/global-pull-secret-syncer-gz9zx"
Sep 09 20:53:28.062422 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:28.062115    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 09 20:53:28.062422 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:28.062115    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 09 20:53:28.079853 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:28.079822    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/732d9d71-d680-4291-aaee-028307ef79dc-original-pull-secret\") pod \"global-pull-secret-syncer-gz9zx\" (UID: \"732d9d71-d680-4291-aaee-028307ef79dc\") " pod="kube-system/global-pull-secret-syncer-gz9zx"
Sep 09 20:53:28.080035 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:28.079829    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c6fd107c-ea2f-4e85-9a32-1e11f0858fd3-metrics-certs\") pod \"network-metrics-daemon-ws8hc\" (UID: \"c6fd107c-ea2f-4e85-9a32-1e11f0858fd3\") " pod="openshift-multus/network-metrics-daemon-ws8hc"
Sep 09 20:53:28.096501 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:28.096470    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gz9zx"
Sep 09 20:53:28.162093 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:28.161964    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6c58d\" (UniqueName: \"kubernetes.io/projected/e1193e63-dea9-41f4-a6ab-748d8ae109e8-kube-api-access-6c58d\") pod \"network-check-target-b6qr7\" (UID: \"e1193e63-dea9-41f4-a6ab-748d8ae109e8\") " pod="openshift-network-diagnostics/network-check-target-b6qr7"
Sep 09 20:53:28.164950 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:28.164737    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 09 20:53:28.180303 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:28.180279    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 09 20:53:28.188110 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:28.188070    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6c58d\" (UniqueName: \"kubernetes.io/projected/e1193e63-dea9-41f4-a6ab-748d8ae109e8-kube-api-access-6c58d\") pod \"network-check-target-b6qr7\" (UID: \"e1193e63-dea9-41f4-a6ab-748d8ae109e8\") " pod="openshift-network-diagnostics/network-check-target-b6qr7"
Sep 09 20:53:28.299693 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:28.299662    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-7m4wj\""
Sep 09 20:53:28.305166 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:28.305142    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-w9kf4\""
Sep 09 20:53:28.308080 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:28.308059    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b6qr7"
Sep 09 20:53:28.313100 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:28.313077    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ws8hc"
Sep 09 20:53:28.392564 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:28.392533    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-gz9zx"]
Sep 09 20:53:28.399828 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:53:28.399763    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod732d9d71_d680_4291_aaee_028307ef79dc.slice/crio-3a32334e136700f294aee42825f7206c87c5944cd42254161e6e620f6546f829 WatchSource:0}: Error finding container 3a32334e136700f294aee42825f7206c87c5944cd42254161e6e620f6546f829: Status 404 returned error can't find the container with id 3a32334e136700f294aee42825f7206c87c5944cd42254161e6e620f6546f829
Sep 09 20:53:28.574999 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:28.574911    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-gz9zx" event={"ID":"732d9d71-d680-4291-aaee-028307ef79dc","Type":"ContainerStarted","Data":"3a32334e136700f294aee42825f7206c87c5944cd42254161e6e620f6546f829"}
Sep 09 20:53:28.610090 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:28.609417    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-b6qr7"]
Sep 09 20:53:28.613908 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:28.613885    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-ws8hc"]
Sep 09 20:53:28.615920 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:53:28.615891    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6fd107c_ea2f_4e85_9a32_1e11f0858fd3.slice/crio-f70fc1173f3a4e286a24cf5a2c082b2f14706052fd9b0ab6e6ec8f6b195d2d2e WatchSource:0}: Error finding container f70fc1173f3a4e286a24cf5a2c082b2f14706052fd9b0ab6e6ec8f6b195d2d2e: Status 404 returned error can't find the container with id f70fc1173f3a4e286a24cf5a2c082b2f14706052fd9b0ab6e6ec8f6b195d2d2e
Sep 09 20:53:28.616577 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:53:28.616555    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1193e63_dea9_41f4_a6ab_748d8ae109e8.slice/crio-56fdcc2cfb5b3d5b59276e7f2085f69bfad0f95a63de9e312b7c71739e57e90a WatchSource:0}: Error finding container 56fdcc2cfb5b3d5b59276e7f2085f69bfad0f95a63de9e312b7c71739e57e90a: Status 404 returned error can't find the container with id 56fdcc2cfb5b3d5b59276e7f2085f69bfad0f95a63de9e312b7c71739e57e90a
Sep 09 20:53:29.580347 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:29.580305    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-b6qr7" event={"ID":"e1193e63-dea9-41f4-a6ab-748d8ae109e8","Type":"ContainerStarted","Data":"56fdcc2cfb5b3d5b59276e7f2085f69bfad0f95a63de9e312b7c71739e57e90a"}
Sep 09 20:53:29.581750 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:29.581717    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-ws8hc" event={"ID":"c6fd107c-ea2f-4e85-9a32-1e11f0858fd3","Type":"ContainerStarted","Data":"f70fc1173f3a4e286a24cf5a2c082b2f14706052fd9b0ab6e6ec8f6b195d2d2e"}
Sep 09 20:53:29.976110 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:29.976060    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d7f530ee-5bfc-4e22-acc2-328625228733-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-gqh2c\" (UID: \"d7f530ee-5bfc-4e22-acc2-328625228733\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-gqh2c"
Sep 09 20:53:29.984317 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:29.984259    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d7f530ee-5bfc-4e22-acc2-328625228733-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-gqh2c\" (UID: \"d7f530ee-5bfc-4e22-acc2-328625228733\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-gqh2c"
Sep 09 20:53:30.220319 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:30.220286    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-vd94f\""
Sep 09 20:53:30.228118 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:30.228050    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-gqh2c"
Sep 09 20:53:31.188689 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:31.188654    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cdcc96965-gqh2c"]
Sep 09 20:53:31.225432 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:53:31.225392    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7f530ee_5bfc_4e22_acc2_328625228733.slice/crio-4ed659a2adce7e2555492f2591b5c26b84bb3f8bf722412dd1da3482da0a2574 WatchSource:0}: Error finding container 4ed659a2adce7e2555492f2591b5c26b84bb3f8bf722412dd1da3482da0a2574: Status 404 returned error can't find the container with id 4ed659a2adce7e2555492f2591b5c26b84bb3f8bf722412dd1da3482da0a2574
Sep 09 20:53:31.594320 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:31.594287    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-ws8hc" event={"ID":"c6fd107c-ea2f-4e85-9a32-1e11f0858fd3","Type":"ContainerStarted","Data":"edcb27fc2a8d8a07b40727486a1ae119ec16bef5dea0f5d651d77db63a186fe9"}
Sep 09 20:53:31.595854 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:31.595826    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-gqh2c" event={"ID":"d7f530ee-5bfc-4e22-acc2-328625228733","Type":"ContainerStarted","Data":"4ed659a2adce7e2555492f2591b5c26b84bb3f8bf722412dd1da3482da0a2574"}
Sep 09 20:53:31.598629 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:31.598601    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7c484d6b5b-s2qds" event={"ID":"255102d9-fa66-4ec7-aa03-f8fab1119cd2","Type":"ContainerStarted","Data":"e1bd8815b5a37ed1d96418b73b418f65f7f45ab02edc0b206666c39b52d9bd4b"}
Sep 09 20:53:31.628754 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:31.628198    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7c484d6b5b-s2qds" podStartSLOduration=2.04242966 podStartE2EDuration="5.628170638s" totalImagesPullingTime="3.585740978s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:53:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:53:31.62661784 +0000 UTC m=+68.866711957" watchObservedRunningTime="2026-09-09 20:53:31.628170638 +0000 UTC m=+68.868264813"
Sep 09 20:53:32.605457 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:32.605398    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-ws8hc" event={"ID":"c6fd107c-ea2f-4e85-9a32-1e11f0858fd3","Type":"ContainerStarted","Data":"2d9dd3a792463134f07f4df0f442029a935ccdbee2cff2f54cb6a5ee4e8d1b2e"}
Sep 09 20:53:32.625163 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:32.622760    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-ws8hc" podStartSLOduration=66.985730637 podStartE2EDuration="1m9.622739648s" totalImagesPullingTime="2.637009011s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:52:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:53:32.621789836 +0000 UTC m=+69.861883963" watchObservedRunningTime="2026-09-09 20:53:32.622739648 +0000 UTC m=+69.862833767"
Sep 09 20:53:35.117778 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:35.117682    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8cb68ecc-5af7-4e01-8032-d41b652555ad-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9z7nf\" (UID: \"8cb68ecc-5af7-4e01-8032-d41b652555ad\") " pod="openshift-insights/insights-runtime-extractor-9z7nf"
Sep 09 20:53:35.120246 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:35.120221    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8cb68ecc-5af7-4e01-8032-d41b652555ad-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9z7nf\" (UID: \"8cb68ecc-5af7-4e01-8032-d41b652555ad\") " pod="openshift-insights/insights-runtime-extractor-9z7nf"
Sep 09 20:53:35.406538 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:35.406512    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-s99rg\""
Sep 09 20:53:35.414579 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:35.414559    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-9z7nf"
Sep 09 20:53:35.619060 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:35.619025    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-gz9zx" event={"ID":"732d9d71-d680-4291-aaee-028307ef79dc","Type":"ContainerStarted","Data":"442e39580dbafba15b4ecc8e28e311cd2a14fe11ad1b923138fe0c98c58e7660"}
Sep 09 20:53:35.620758 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:35.620729    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-b6qr7" event={"ID":"e1193e63-dea9-41f4-a6ab-748d8ae109e8","Type":"ContainerStarted","Data":"15f188a68702c42289d2f6386ad79c3741698e5059c5ff57d2e499ba52703ad8"}
Sep 09 20:53:35.620890 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:35.620786    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-b6qr7"
Sep 09 20:53:35.622350 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:35.622330    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-gqh2c" event={"ID":"d7f530ee-5bfc-4e22-acc2-328625228733","Type":"ContainerStarted","Data":"d1bc3a070ab0e417fc3eaebc8472bc68f7763d6005193818b4ed69a4dc29f208"}
Sep 09 20:53:35.642747 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:35.642694    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-gz9zx" podStartSLOduration=66.205082629 podStartE2EDuration="1m12.642676874s" totalImagesPullingTime="6.437594245s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:52:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:53:35.642397197 +0000 UTC m=+72.882491315" watchObservedRunningTime="2026-09-09 20:53:35.642676874 +0000 UTC m=+72.882770991"
Sep 09 20:53:35.666233 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:35.666121    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-gqh2c" podStartSLOduration=34.069333437 podStartE2EDuration="37.666105451s" totalImagesPullingTime="3.596772014s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:52:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:53:35.665630828 +0000 UTC m=+72.905724959" watchObservedRunningTime="2026-09-09 20:53:35.666105451 +0000 UTC m=+72.906199569"
Sep 09 20:53:35.692989 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:35.692956    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-9z7nf"]
Sep 09 20:53:35.696537 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:53:35.696512    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8cb68ecc_5af7_4e01_8032_d41b652555ad.slice/crio-fc392f88d1a462404b6af8c034950560aa02f4395bba6ece0ad97cf5ea07b250 WatchSource:0}: Error finding container fc392f88d1a462404b6af8c034950560aa02f4395bba6ece0ad97cf5ea07b250: Status 404 returned error can't find the container with id fc392f88d1a462404b6af8c034950560aa02f4395bba6ece0ad97cf5ea07b250
Sep 09 20:53:35.697205 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:35.697163    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-b6qr7" podStartSLOduration=66.467016748 podStartE2EDuration="1m12.69714812s" totalImagesPullingTime="6.230131372s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:52:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:53:35.694944657 +0000 UTC m=+72.935038775" watchObservedRunningTime="2026-09-09 20:53:35.69714812 +0000 UTC m=+72.937242239"
Sep 09 20:53:36.628058 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:36.627970    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9z7nf" event={"ID":"8cb68ecc-5af7-4e01-8032-d41b652555ad","Type":"ContainerStarted","Data":"0ceda898a0c8e4d69f0f6241a6f0f04c6666d38b622b3da10d4f603c3aae0d60"}
Sep 09 20:53:36.628058 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:36.628041    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9z7nf" event={"ID":"8cb68ecc-5af7-4e01-8032-d41b652555ad","Type":"ContainerStarted","Data":"c8b53566f7e1d486828f64607957663551eaf8313f06a6221424b5bbed93c899"}
Sep 09 20:53:36.628428 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:36.628071    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9z7nf" event={"ID":"8cb68ecc-5af7-4e01-8032-d41b652555ad","Type":"ContainerStarted","Data":"fc392f88d1a462404b6af8c034950560aa02f4395bba6ece0ad97cf5ea07b250"}
Sep 09 20:53:36.695182 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:36.695153    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-64bb684d65-d9qxp"
Sep 09 20:53:38.636797 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:38.636763    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9z7nf" event={"ID":"8cb68ecc-5af7-4e01-8032-d41b652555ad","Type":"ContainerStarted","Data":"df15f2a9dc7d772281ffdbf48923772441848a84fc3c5e70a97963ecb8fd4247"}
Sep 09 20:53:38.668799 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:38.668752    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-9z7nf" podStartSLOduration=33.798890715 podStartE2EDuration="35.668734023s" totalImagesPullingTime="1.869843308s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:53:03 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:53:38.667655483 +0000 UTC m=+75.907749583" watchObservedRunningTime="2026-09-09 20:53:38.668734023 +0000 UTC m=+75.908828140"
Sep 09 20:53:42.426745 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:42.426710    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-655476f677-9tncl"]
Sep 09 20:53:42.429897 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:42.429880    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-655476f677-9tncl"
Sep 09 20:53:42.436725 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:42.436703    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-jzlgf\""
Sep 09 20:53:42.436873 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:42.436708    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 09 20:53:42.436873 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:42.436708    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 09 20:53:42.436873 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:42.436705    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 09 20:53:42.437419 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:42.437399    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 09 20:53:42.437843 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:42.437827    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 09 20:53:42.437843 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:42.437840    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 09 20:53:42.438014 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:42.437898    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 09 20:53:42.440478 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:42.440461    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 09 20:53:42.444029 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:42.444009    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-655476f677-9tncl"]
Sep 09 20:53:42.474373 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:42.474341    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f66ww\" (UniqueName: \"kubernetes.io/projected/f4343b3e-2a6c-41f8-95df-f9a50bcc3d8d-kube-api-access-f66ww\") pod \"console-655476f677-9tncl\" (UID: \"f4343b3e-2a6c-41f8-95df-f9a50bcc3d8d\") " pod="openshift-console/console-655476f677-9tncl"
Sep 09 20:53:42.474373 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:42.474380    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f4343b3e-2a6c-41f8-95df-f9a50bcc3d8d-console-oauth-config\") pod \"console-655476f677-9tncl\" (UID: \"f4343b3e-2a6c-41f8-95df-f9a50bcc3d8d\") " pod="openshift-console/console-655476f677-9tncl"
Sep 09 20:53:42.474635 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:42.474425    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f4343b3e-2a6c-41f8-95df-f9a50bcc3d8d-oauth-serving-cert\") pod \"console-655476f677-9tncl\" (UID: \"f4343b3e-2a6c-41f8-95df-f9a50bcc3d8d\") " pod="openshift-console/console-655476f677-9tncl"
Sep 09 20:53:42.474635 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:42.474490    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f4343b3e-2a6c-41f8-95df-f9a50bcc3d8d-console-config\") pod \"console-655476f677-9tncl\" (UID: \"f4343b3e-2a6c-41f8-95df-f9a50bcc3d8d\") " pod="openshift-console/console-655476f677-9tncl"
Sep 09 20:53:42.474635 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:42.474515    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f4343b3e-2a6c-41f8-95df-f9a50bcc3d8d-service-ca\") pod \"console-655476f677-9tncl\" (UID: \"f4343b3e-2a6c-41f8-95df-f9a50bcc3d8d\") " pod="openshift-console/console-655476f677-9tncl"
Sep 09 20:53:42.474635 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:42.474546    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f4343b3e-2a6c-41f8-95df-f9a50bcc3d8d-trusted-ca-bundle\") pod \"console-655476f677-9tncl\" (UID: \"f4343b3e-2a6c-41f8-95df-f9a50bcc3d8d\") " pod="openshift-console/console-655476f677-9tncl"
Sep 09 20:53:42.474635 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:42.474576    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f4343b3e-2a6c-41f8-95df-f9a50bcc3d8d-console-serving-cert\") pod \"console-655476f677-9tncl\" (UID: \"f4343b3e-2a6c-41f8-95df-f9a50bcc3d8d\") " pod="openshift-console/console-655476f677-9tncl"
Sep 09 20:53:42.575429 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:42.575389    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f4343b3e-2a6c-41f8-95df-f9a50bcc3d8d-oauth-serving-cert\") pod \"console-655476f677-9tncl\" (UID: \"f4343b3e-2a6c-41f8-95df-f9a50bcc3d8d\") " pod="openshift-console/console-655476f677-9tncl"
Sep 09 20:53:42.575588 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:42.575434    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f4343b3e-2a6c-41f8-95df-f9a50bcc3d8d-console-config\") pod \"console-655476f677-9tncl\" (UID: \"f4343b3e-2a6c-41f8-95df-f9a50bcc3d8d\") " pod="openshift-console/console-655476f677-9tncl"
Sep 09 20:53:42.575588 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:42.575490    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f4343b3e-2a6c-41f8-95df-f9a50bcc3d8d-service-ca\") pod \"console-655476f677-9tncl\" (UID: \"f4343b3e-2a6c-41f8-95df-f9a50bcc3d8d\") " pod="openshift-console/console-655476f677-9tncl"
Sep 09 20:53:42.575588 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:42.575514    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f4343b3e-2a6c-41f8-95df-f9a50bcc3d8d-trusted-ca-bundle\") pod \"console-655476f677-9tncl\" (UID: \"f4343b3e-2a6c-41f8-95df-f9a50bcc3d8d\") " pod="openshift-console/console-655476f677-9tncl"
Sep 09 20:53:42.575588 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:42.575544    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f4343b3e-2a6c-41f8-95df-f9a50bcc3d8d-console-serving-cert\") pod \"console-655476f677-9tncl\" (UID: \"f4343b3e-2a6c-41f8-95df-f9a50bcc3d8d\") " pod="openshift-console/console-655476f677-9tncl"
Sep 09 20:53:42.575812 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:42.575604    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f66ww\" (UniqueName: \"kubernetes.io/projected/f4343b3e-2a6c-41f8-95df-f9a50bcc3d8d-kube-api-access-f66ww\") pod \"console-655476f677-9tncl\" (UID: \"f4343b3e-2a6c-41f8-95df-f9a50bcc3d8d\") " pod="openshift-console/console-655476f677-9tncl"
Sep 09 20:53:42.575812 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:42.575632    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f4343b3e-2a6c-41f8-95df-f9a50bcc3d8d-console-oauth-config\") pod \"console-655476f677-9tncl\" (UID: \"f4343b3e-2a6c-41f8-95df-f9a50bcc3d8d\") " pod="openshift-console/console-655476f677-9tncl"
Sep 09 20:53:42.576239 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:42.576209    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f4343b3e-2a6c-41f8-95df-f9a50bcc3d8d-console-config\") pod \"console-655476f677-9tncl\" (UID: \"f4343b3e-2a6c-41f8-95df-f9a50bcc3d8d\") " pod="openshift-console/console-655476f677-9tncl"
Sep 09 20:53:42.576374 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:42.576352    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f4343b3e-2a6c-41f8-95df-f9a50bcc3d8d-oauth-serving-cert\") pod \"console-655476f677-9tncl\" (UID: \"f4343b3e-2a6c-41f8-95df-f9a50bcc3d8d\") " pod="openshift-console/console-655476f677-9tncl"
Sep 09 20:53:42.576497 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:42.576469    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f4343b3e-2a6c-41f8-95df-f9a50bcc3d8d-trusted-ca-bundle\") pod \"console-655476f677-9tncl\" (UID: \"f4343b3e-2a6c-41f8-95df-f9a50bcc3d8d\") " pod="openshift-console/console-655476f677-9tncl"
Sep 09 20:53:42.576559 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:42.576428    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f4343b3e-2a6c-41f8-95df-f9a50bcc3d8d-service-ca\") pod \"console-655476f677-9tncl\" (UID: \"f4343b3e-2a6c-41f8-95df-f9a50bcc3d8d\") " pod="openshift-console/console-655476f677-9tncl"
Sep 09 20:53:42.578094 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:42.578070    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f4343b3e-2a6c-41f8-95df-f9a50bcc3d8d-console-serving-cert\") pod \"console-655476f677-9tncl\" (UID: \"f4343b3e-2a6c-41f8-95df-f9a50bcc3d8d\") " pod="openshift-console/console-655476f677-9tncl"
Sep 09 20:53:42.578188 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:42.578171    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f4343b3e-2a6c-41f8-95df-f9a50bcc3d8d-console-oauth-config\") pod \"console-655476f677-9tncl\" (UID: \"f4343b3e-2a6c-41f8-95df-f9a50bcc3d8d\") " pod="openshift-console/console-655476f677-9tncl"
Sep 09 20:53:42.585029 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:42.585009    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f66ww\" (UniqueName: \"kubernetes.io/projected/f4343b3e-2a6c-41f8-95df-f9a50bcc3d8d-kube-api-access-f66ww\") pod \"console-655476f677-9tncl\" (UID: \"f4343b3e-2a6c-41f8-95df-f9a50bcc3d8d\") " pod="openshift-console/console-655476f677-9tncl"
Sep 09 20:53:42.739426 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:42.739328    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-655476f677-9tncl"
Sep 09 20:53:43.013650 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:43.013567    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-655476f677-9tncl"]
Sep 09 20:53:43.016177 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:53:43.016141    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4343b3e_2a6c_41f8_95df_f9a50bcc3d8d.slice/crio-a0a630da8c04f76fc5da9fd061388a641b7d07248898782e360e81803b123b7e WatchSource:0}: Error finding container a0a630da8c04f76fc5da9fd061388a641b7d07248898782e360e81803b123b7e: Status 404 returned error can't find the container with id a0a630da8c04f76fc5da9fd061388a641b7d07248898782e360e81803b123b7e
Sep 09 20:53:43.655140 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:43.655101    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-655476f677-9tncl" event={"ID":"f4343b3e-2a6c-41f8-95df-f9a50bcc3d8d","Type":"ContainerStarted","Data":"a0a630da8c04f76fc5da9fd061388a641b7d07248898782e360e81803b123b7e"}
Sep 09 20:53:46.666285 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:46.666250    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-655476f677-9tncl" event={"ID":"f4343b3e-2a6c-41f8-95df-f9a50bcc3d8d","Type":"ContainerStarted","Data":"dfe84bc04576025a0f854e68a34f3207533f4b1f5fd2f7284f926443889a2b2b"}
Sep 09 20:53:46.695236 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:46.695187    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-655476f677-9tncl" podStartSLOduration=2.054234128 podStartE2EDuration="4.695171401s" totalImagesPullingTime="2.640937273s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:53:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:53:46.694267178 +0000 UTC m=+83.934361313" watchObservedRunningTime="2026-09-09 20:53:46.695171401 +0000 UTC m=+83.935265518"
Sep 09 20:53:48.579159 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:48.579130    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6f4bd54dd8-fqmrm"
Sep 09 20:53:51.709574 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:51.709531    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-64bb684d65-d9qxp" podUID="729f090c-e6f4-4757-98ec-9eba867c9ca8" containerName="registry" containerID="cri-o://3e1bbc2bdd6373de148fbea32c88b0e090045f1730a0dfd2ead524a687abb8d1" gracePeriod=30
Sep 09 20:53:51.940394 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:51.940372    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-64bb684d65-d9qxp"
Sep 09 20:53:52.045305 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:52.045238    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/729f090c-e6f4-4757-98ec-9eba867c9ca8-bound-sa-token\") pod \"729f090c-e6f4-4757-98ec-9eba867c9ca8\" (UID: \"729f090c-e6f4-4757-98ec-9eba867c9ca8\") "
Sep 09 20:53:52.045305 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:52.045273    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/729f090c-e6f4-4757-98ec-9eba867c9ca8-installation-pull-secrets\") pod \"729f090c-e6f4-4757-98ec-9eba867c9ca8\" (UID: \"729f090c-e6f4-4757-98ec-9eba867c9ca8\") "
Sep 09 20:53:52.045305 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:52.045303    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kt8r9\" (UniqueName: \"kubernetes.io/projected/729f090c-e6f4-4757-98ec-9eba867c9ca8-kube-api-access-kt8r9\") pod \"729f090c-e6f4-4757-98ec-9eba867c9ca8\" (UID: \"729f090c-e6f4-4757-98ec-9eba867c9ca8\") "
Sep 09 20:53:52.045559 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:52.045322    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/729f090c-e6f4-4757-98ec-9eba867c9ca8-trusted-ca\") pod \"729f090c-e6f4-4757-98ec-9eba867c9ca8\" (UID: \"729f090c-e6f4-4757-98ec-9eba867c9ca8\") "
Sep 09 20:53:52.045559 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:52.045342    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/729f090c-e6f4-4757-98ec-9eba867c9ca8-image-registry-private-configuration\") pod \"729f090c-e6f4-4757-98ec-9eba867c9ca8\" (UID: \"729f090c-e6f4-4757-98ec-9eba867c9ca8\") "
Sep 09 20:53:52.045559 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:52.045367    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/729f090c-e6f4-4757-98ec-9eba867c9ca8-registry-certificates\") pod \"729f090c-e6f4-4757-98ec-9eba867c9ca8\" (UID: \"729f090c-e6f4-4757-98ec-9eba867c9ca8\") "
Sep 09 20:53:52.045559 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:52.045419    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/729f090c-e6f4-4757-98ec-9eba867c9ca8-ca-trust-extracted\") pod \"729f090c-e6f4-4757-98ec-9eba867c9ca8\" (UID: \"729f090c-e6f4-4757-98ec-9eba867c9ca8\") "
Sep 09 20:53:52.045559 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:52.045462    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/729f090c-e6f4-4757-98ec-9eba867c9ca8-registry-tls\") pod \"729f090c-e6f4-4757-98ec-9eba867c9ca8\" (UID: \"729f090c-e6f4-4757-98ec-9eba867c9ca8\") "
Sep 09 20:53:52.045873 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:52.045845    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/729f090c-e6f4-4757-98ec-9eba867c9ca8-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "729f090c-e6f4-4757-98ec-9eba867c9ca8" (UID: "729f090c-e6f4-4757-98ec-9eba867c9ca8"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:53:52.046142 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:52.045989    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/729f090c-e6f4-4757-98ec-9eba867c9ca8-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "729f090c-e6f4-4757-98ec-9eba867c9ca8" (UID: "729f090c-e6f4-4757-98ec-9eba867c9ca8"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:53:52.048346 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:52.048311    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/729f090c-e6f4-4757-98ec-9eba867c9ca8-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "729f090c-e6f4-4757-98ec-9eba867c9ca8" (UID: "729f090c-e6f4-4757-98ec-9eba867c9ca8"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:53:52.048470 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:52.048388    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/729f090c-e6f4-4757-98ec-9eba867c9ca8-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "729f090c-e6f4-4757-98ec-9eba867c9ca8" (UID: "729f090c-e6f4-4757-98ec-9eba867c9ca8"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:53:52.048544 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:52.048464    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/729f090c-e6f4-4757-98ec-9eba867c9ca8-kube-api-access-kt8r9" (OuterVolumeSpecName: "kube-api-access-kt8r9") pod "729f090c-e6f4-4757-98ec-9eba867c9ca8" (UID: "729f090c-e6f4-4757-98ec-9eba867c9ca8"). InnerVolumeSpecName "kube-api-access-kt8r9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:53:52.048544 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:52.048534    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/729f090c-e6f4-4757-98ec-9eba867c9ca8-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "729f090c-e6f4-4757-98ec-9eba867c9ca8" (UID: "729f090c-e6f4-4757-98ec-9eba867c9ca8"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:53:52.048639 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:52.048601    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/729f090c-e6f4-4757-98ec-9eba867c9ca8-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "729f090c-e6f4-4757-98ec-9eba867c9ca8" (UID: "729f090c-e6f4-4757-98ec-9eba867c9ca8"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:53:52.054626 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:52.054604    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/729f090c-e6f4-4757-98ec-9eba867c9ca8-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "729f090c-e6f4-4757-98ec-9eba867c9ca8" (UID: "729f090c-e6f4-4757-98ec-9eba867c9ca8"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:53:52.146616 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:52.146593    2579 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/729f090c-e6f4-4757-98ec-9eba867c9ca8-bound-sa-token\") on node \"ip-10-0-138-224.ec2.internal\" DevicePath \"\""
Sep 09 20:53:52.146616 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:52.146614    2579 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/729f090c-e6f4-4757-98ec-9eba867c9ca8-installation-pull-secrets\") on node \"ip-10-0-138-224.ec2.internal\" DevicePath \"\""
Sep 09 20:53:52.146727 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:52.146625    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kt8r9\" (UniqueName: \"kubernetes.io/projected/729f090c-e6f4-4757-98ec-9eba867c9ca8-kube-api-access-kt8r9\") on node \"ip-10-0-138-224.ec2.internal\" DevicePath \"\""
Sep 09 20:53:52.146727 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:52.146634    2579 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/729f090c-e6f4-4757-98ec-9eba867c9ca8-trusted-ca\") on node \"ip-10-0-138-224.ec2.internal\" DevicePath \"\""
Sep 09 20:53:52.146727 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:52.146643    2579 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/729f090c-e6f4-4757-98ec-9eba867c9ca8-image-registry-private-configuration\") on node \"ip-10-0-138-224.ec2.internal\" DevicePath \"\""
Sep 09 20:53:52.146727 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:52.146652    2579 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/729f090c-e6f4-4757-98ec-9eba867c9ca8-registry-certificates\") on node \"ip-10-0-138-224.ec2.internal\" DevicePath \"\""
Sep 09 20:53:52.146727 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:52.146660    2579 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/729f090c-e6f4-4757-98ec-9eba867c9ca8-ca-trust-extracted\") on node \"ip-10-0-138-224.ec2.internal\" DevicePath \"\""
Sep 09 20:53:52.146727 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:52.146668    2579 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/729f090c-e6f4-4757-98ec-9eba867c9ca8-registry-tls\") on node \"ip-10-0-138-224.ec2.internal\" DevicePath \"\""
Sep 09 20:53:52.688652 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:52.688614    2579 generic.go:356] "Generic (PLEG): container finished" podID="729f090c-e6f4-4757-98ec-9eba867c9ca8" containerID="3e1bbc2bdd6373de148fbea32c88b0e090045f1730a0dfd2ead524a687abb8d1" exitCode=0
Sep 09 20:53:52.688863 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:52.688664    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-64bb684d65-d9qxp"
Sep 09 20:53:52.688863 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:52.688691    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-64bb684d65-d9qxp" event={"ID":"729f090c-e6f4-4757-98ec-9eba867c9ca8","Type":"ContainerDied","Data":"3e1bbc2bdd6373de148fbea32c88b0e090045f1730a0dfd2ead524a687abb8d1"}
Sep 09 20:53:52.688863 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:52.688741    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-64bb684d65-d9qxp" event={"ID":"729f090c-e6f4-4757-98ec-9eba867c9ca8","Type":"ContainerDied","Data":"88d8ade4ba59262d8b35b8eca4aed1e85643c86efce04c2b89e3d4558f155576"}
Sep 09 20:53:52.688863 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:52.688766    2579 scope.go:117] "RemoveContainer" containerID="3e1bbc2bdd6373de148fbea32c88b0e090045f1730a0dfd2ead524a687abb8d1"
Sep 09 20:53:52.697981 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:52.697958    2579 scope.go:117] "RemoveContainer" containerID="3e1bbc2bdd6373de148fbea32c88b0e090045f1730a0dfd2ead524a687abb8d1"
Sep 09 20:53:52.698271 ip-10-0-138-224 kubenswrapper[2579]: E0909 20:53:52.698244    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e1bbc2bdd6373de148fbea32c88b0e090045f1730a0dfd2ead524a687abb8d1\": container with ID starting with 3e1bbc2bdd6373de148fbea32c88b0e090045f1730a0dfd2ead524a687abb8d1 not found: ID does not exist" containerID="3e1bbc2bdd6373de148fbea32c88b0e090045f1730a0dfd2ead524a687abb8d1"
Sep 09 20:53:52.698322 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:52.698282    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e1bbc2bdd6373de148fbea32c88b0e090045f1730a0dfd2ead524a687abb8d1"} err="failed to get container status \"3e1bbc2bdd6373de148fbea32c88b0e090045f1730a0dfd2ead524a687abb8d1\": rpc error: code = NotFound desc = could not find container \"3e1bbc2bdd6373de148fbea32c88b0e090045f1730a0dfd2ead524a687abb8d1\": container with ID starting with 3e1bbc2bdd6373de148fbea32c88b0e090045f1730a0dfd2ead524a687abb8d1 not found: ID does not exist"
Sep 09 20:53:52.710964 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:52.710944    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-64bb684d65-d9qxp"]
Sep 09 20:53:52.712774 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:52.712753    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-64bb684d65-d9qxp"]
Sep 09 20:53:52.740338 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:52.740318    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-655476f677-9tncl"
Sep 09 20:53:52.740415 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:52.740374    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-655476f677-9tncl"
Sep 09 20:53:52.745064 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:52.745048    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-655476f677-9tncl"
Sep 09 20:53:53.289682 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:53.289646    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="729f090c-e6f4-4757-98ec-9eba867c9ca8" path="/var/lib/kubelet/pods/729f090c-e6f4-4757-98ec-9eba867c9ca8/volumes"
Sep 09 20:53:53.697181 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:53.697150    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-655476f677-9tncl"
Sep 09 20:53:53.926985 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:53.926952    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-657fd6c54-84zfh"]
Sep 09 20:53:53.927365 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:53.927177    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="729f090c-e6f4-4757-98ec-9eba867c9ca8" containerName="registry"
Sep 09 20:53:53.927365 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:53.927188    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="729f090c-e6f4-4757-98ec-9eba867c9ca8" containerName="registry"
Sep 09 20:53:53.927365 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:53.927234    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="729f090c-e6f4-4757-98ec-9eba867c9ca8" containerName="registry"
Sep 09 20:53:53.931998 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:53.931981    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-84zfh"
Sep 09 20:53:53.935382 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:53.935362    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Sep 09 20:53:53.935705 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:53.935683    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-l6bw8\""
Sep 09 20:53:53.935705 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:53.935692    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 09 20:53:53.935833 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:53.935692    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Sep 09 20:53:53.943710 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:53.943691    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-657fd6c54-84zfh"]
Sep 09 20:53:53.987630 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:53.987573    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-8lr9d"]
Sep 09 20:53:53.990511 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:53.990496    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-8lr9d"
Sep 09 20:53:53.992932 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:53.992915    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 09 20:53:53.993010 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:53.992954    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 09 20:53:53.993111 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:53.993095    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-v2wxl\""
Sep 09 20:53:53.993191 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:53.993175    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 09 20:53:54.058992 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:54.058971    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ebe524c8-2574-47dd-864c-b558de801715-root\") pod \"node-exporter-8lr9d\" (UID: \"ebe524c8-2574-47dd-864c-b558de801715\") " pod="openshift-monitoring/node-exporter-8lr9d"
Sep 09 20:53:54.059100 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:54.059001    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8750f8b1-ad24-45b8-b483-8bc96f4d6198-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-657fd6c54-84zfh\" (UID: \"8750f8b1-ad24-45b8-b483-8bc96f4d6198\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-84zfh"
Sep 09 20:53:54.059100 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:54.059027    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9gfhr\" (UniqueName: \"kubernetes.io/projected/ebe524c8-2574-47dd-864c-b558de801715-kube-api-access-9gfhr\") pod \"node-exporter-8lr9d\" (UID: \"ebe524c8-2574-47dd-864c-b558de801715\") " pod="openshift-monitoring/node-exporter-8lr9d"
Sep 09 20:53:54.059100 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:54.059078    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ebe524c8-2574-47dd-864c-b558de801715-node-exporter-wtmp\") pod \"node-exporter-8lr9d\" (UID: \"ebe524c8-2574-47dd-864c-b558de801715\") " pod="openshift-monitoring/node-exporter-8lr9d"
Sep 09 20:53:54.059240 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:54.059125    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ebe524c8-2574-47dd-864c-b558de801715-sys\") pod \"node-exporter-8lr9d\" (UID: \"ebe524c8-2574-47dd-864c-b558de801715\") " pod="openshift-monitoring/node-exporter-8lr9d"
Sep 09 20:53:54.059240 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:54.059163    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8750f8b1-ad24-45b8-b483-8bc96f4d6198-metrics-client-ca\") pod \"openshift-state-metrics-657fd6c54-84zfh\" (UID: \"8750f8b1-ad24-45b8-b483-8bc96f4d6198\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-84zfh"
Sep 09 20:53:54.059240 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:54.059192    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/8750f8b1-ad24-45b8-b483-8bc96f4d6198-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-84zfh\" (UID: \"8750f8b1-ad24-45b8-b483-8bc96f4d6198\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-84zfh"
Sep 09 20:53:54.059240 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:54.059231    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ebe524c8-2574-47dd-864c-b558de801715-node-exporter-tls\") pod \"node-exporter-8lr9d\" (UID: \"ebe524c8-2574-47dd-864c-b558de801715\") " pod="openshift-monitoring/node-exporter-8lr9d"
Sep 09 20:53:54.059385 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:54.059254    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ebe524c8-2574-47dd-864c-b558de801715-node-exporter-accelerators-collector-config\") pod \"node-exporter-8lr9d\" (UID: \"ebe524c8-2574-47dd-864c-b558de801715\") " pod="openshift-monitoring/node-exporter-8lr9d"
Sep 09 20:53:54.059385 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:54.059286    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ebe524c8-2574-47dd-864c-b558de801715-metrics-client-ca\") pod \"node-exporter-8lr9d\" (UID: \"ebe524c8-2574-47dd-864c-b558de801715\") " pod="openshift-monitoring/node-exporter-8lr9d"
Sep 09 20:53:54.059385 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:54.059318    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ebe524c8-2574-47dd-864c-b558de801715-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8lr9d\" (UID: \"ebe524c8-2574-47dd-864c-b558de801715\") " pod="openshift-monitoring/node-exporter-8lr9d"
Sep 09 20:53:54.059385 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:54.059350    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6jtt\" (UniqueName: \"kubernetes.io/projected/8750f8b1-ad24-45b8-b483-8bc96f4d6198-kube-api-access-z6jtt\") pod \"openshift-state-metrics-657fd6c54-84zfh\" (UID: \"8750f8b1-ad24-45b8-b483-8bc96f4d6198\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-84zfh"
Sep 09 20:53:54.059561 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:54.059406    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ebe524c8-2574-47dd-864c-b558de801715-node-exporter-textfile\") pod \"node-exporter-8lr9d\" (UID: \"ebe524c8-2574-47dd-864c-b558de801715\") " pod="openshift-monitoring/node-exporter-8lr9d"
Sep 09 20:53:54.159914 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:54.159890    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ebe524c8-2574-47dd-864c-b558de801715-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8lr9d\" (UID: \"ebe524c8-2574-47dd-864c-b558de801715\") " pod="openshift-monitoring/node-exporter-8lr9d"
Sep 09 20:53:54.160052 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:54.159920    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6jtt\" (UniqueName: \"kubernetes.io/projected/8750f8b1-ad24-45b8-b483-8bc96f4d6198-kube-api-access-z6jtt\") pod \"openshift-state-metrics-657fd6c54-84zfh\" (UID: \"8750f8b1-ad24-45b8-b483-8bc96f4d6198\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-84zfh"
Sep 09 20:53:54.160052 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:54.159941    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ebe524c8-2574-47dd-864c-b558de801715-node-exporter-textfile\") pod \"node-exporter-8lr9d\" (UID: \"ebe524c8-2574-47dd-864c-b558de801715\") " pod="openshift-monitoring/node-exporter-8lr9d"
Sep 09 20:53:54.160052 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:54.159959    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ebe524c8-2574-47dd-864c-b558de801715-root\") pod \"node-exporter-8lr9d\" (UID: \"ebe524c8-2574-47dd-864c-b558de801715\") " pod="openshift-monitoring/node-exporter-8lr9d"
Sep 09 20:53:54.160052 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:54.159982    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8750f8b1-ad24-45b8-b483-8bc96f4d6198-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-657fd6c54-84zfh\" (UID: \"8750f8b1-ad24-45b8-b483-8bc96f4d6198\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-84zfh"
Sep 09 20:53:54.160052 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:54.160035    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ebe524c8-2574-47dd-864c-b558de801715-root\") pod \"node-exporter-8lr9d\" (UID: \"ebe524c8-2574-47dd-864c-b558de801715\") " pod="openshift-monitoring/node-exporter-8lr9d"
Sep 09 20:53:54.160353 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:54.160104    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9gfhr\" (UniqueName: \"kubernetes.io/projected/ebe524c8-2574-47dd-864c-b558de801715-kube-api-access-9gfhr\") pod \"node-exporter-8lr9d\" (UID: \"ebe524c8-2574-47dd-864c-b558de801715\") " pod="openshift-monitoring/node-exporter-8lr9d"
Sep 09 20:53:54.160353 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:54.160146    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ebe524c8-2574-47dd-864c-b558de801715-node-exporter-wtmp\") pod \"node-exporter-8lr9d\" (UID: \"ebe524c8-2574-47dd-864c-b558de801715\") " pod="openshift-monitoring/node-exporter-8lr9d"
Sep 09 20:53:54.160353 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:54.160188    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ebe524c8-2574-47dd-864c-b558de801715-sys\") pod \"node-exporter-8lr9d\" (UID: \"ebe524c8-2574-47dd-864c-b558de801715\") " pod="openshift-monitoring/node-exporter-8lr9d"
Sep 09 20:53:54.160353 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:54.160213    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8750f8b1-ad24-45b8-b483-8bc96f4d6198-metrics-client-ca\") pod \"openshift-state-metrics-657fd6c54-84zfh\" (UID: \"8750f8b1-ad24-45b8-b483-8bc96f4d6198\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-84zfh"
Sep 09 20:53:54.160353 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:54.160247    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/8750f8b1-ad24-45b8-b483-8bc96f4d6198-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-84zfh\" (UID: \"8750f8b1-ad24-45b8-b483-8bc96f4d6198\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-84zfh"
Sep 09 20:53:54.160353 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:54.160260    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ebe524c8-2574-47dd-864c-b558de801715-sys\") pod \"node-exporter-8lr9d\" (UID: \"ebe524c8-2574-47dd-864c-b558de801715\") " pod="openshift-monitoring/node-exporter-8lr9d"
Sep 09 20:53:54.160353 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:54.160273    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ebe524c8-2574-47dd-864c-b558de801715-node-exporter-tls\") pod \"node-exporter-8lr9d\" (UID: \"ebe524c8-2574-47dd-864c-b558de801715\") " pod="openshift-monitoring/node-exporter-8lr9d"
Sep 09 20:53:54.160353 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:54.160314    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ebe524c8-2574-47dd-864c-b558de801715-node-exporter-accelerators-collector-config\") pod \"node-exporter-8lr9d\" (UID: \"ebe524c8-2574-47dd-864c-b558de801715\") " pod="openshift-monitoring/node-exporter-8lr9d"
Sep 09 20:53:54.160353 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:54.160340    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ebe524c8-2574-47dd-864c-b558de801715-node-exporter-wtmp\") pod \"node-exporter-8lr9d\" (UID: \"ebe524c8-2574-47dd-864c-b558de801715\") " pod="openshift-monitoring/node-exporter-8lr9d"
Sep 09 20:53:54.160353 ip-10-0-138-224 kubenswrapper[2579]: E0909 20:53:54.160353    2579 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 09 20:53:54.161044 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:54.160373    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ebe524c8-2574-47dd-864c-b558de801715-metrics-client-ca\") pod \"node-exporter-8lr9d\" (UID: \"ebe524c8-2574-47dd-864c-b558de801715\") " pod="openshift-monitoring/node-exporter-8lr9d"
Sep 09 20:53:54.161044 ip-10-0-138-224 kubenswrapper[2579]: E0909 20:53:54.160408    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ebe524c8-2574-47dd-864c-b558de801715-node-exporter-tls podName:ebe524c8-2574-47dd-864c-b558de801715 nodeName:}" failed. No retries permitted until 2026-09-09 20:53:54.660389055 +0000 UTC m=+91.900483170 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/ebe524c8-2574-47dd-864c-b558de801715-node-exporter-tls") pod "node-exporter-8lr9d" (UID: "ebe524c8-2574-47dd-864c-b558de801715") : secret "node-exporter-tls" not found
Sep 09 20:53:54.161165 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:54.161066    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8750f8b1-ad24-45b8-b483-8bc96f4d6198-metrics-client-ca\") pod \"openshift-state-metrics-657fd6c54-84zfh\" (UID: \"8750f8b1-ad24-45b8-b483-8bc96f4d6198\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-84zfh"
Sep 09 20:53:54.161224 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:54.161194    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ebe524c8-2574-47dd-864c-b558de801715-metrics-client-ca\") pod \"node-exporter-8lr9d\" (UID: \"ebe524c8-2574-47dd-864c-b558de801715\") " pod="openshift-monitoring/node-exporter-8lr9d"
Sep 09 20:53:54.161280 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:54.161236    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ebe524c8-2574-47dd-864c-b558de801715-node-exporter-accelerators-collector-config\") pod \"node-exporter-8lr9d\" (UID: \"ebe524c8-2574-47dd-864c-b558de801715\") " pod="openshift-monitoring/node-exporter-8lr9d"
Sep 09 20:53:54.161330 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:54.161298    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ebe524c8-2574-47dd-864c-b558de801715-node-exporter-textfile\") pod \"node-exporter-8lr9d\" (UID: \"ebe524c8-2574-47dd-864c-b558de801715\") " pod="openshift-monitoring/node-exporter-8lr9d"
Sep 09 20:53:54.162579 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:54.162558    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ebe524c8-2574-47dd-864c-b558de801715-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8lr9d\" (UID: \"ebe524c8-2574-47dd-864c-b558de801715\") " pod="openshift-monitoring/node-exporter-8lr9d"
Sep 09 20:53:54.162687 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:54.162666    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8750f8b1-ad24-45b8-b483-8bc96f4d6198-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-657fd6c54-84zfh\" (UID: \"8750f8b1-ad24-45b8-b483-8bc96f4d6198\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-84zfh"
Sep 09 20:53:54.162782 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:54.162761    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/8750f8b1-ad24-45b8-b483-8bc96f4d6198-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-84zfh\" (UID: \"8750f8b1-ad24-45b8-b483-8bc96f4d6198\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-84zfh"
Sep 09 20:53:54.201117 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:54.201095    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6jtt\" (UniqueName: \"kubernetes.io/projected/8750f8b1-ad24-45b8-b483-8bc96f4d6198-kube-api-access-z6jtt\") pod \"openshift-state-metrics-657fd6c54-84zfh\" (UID: \"8750f8b1-ad24-45b8-b483-8bc96f4d6198\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-84zfh"
Sep 09 20:53:54.235877 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:54.235855    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9gfhr\" (UniqueName: \"kubernetes.io/projected/ebe524c8-2574-47dd-864c-b558de801715-kube-api-access-9gfhr\") pod \"node-exporter-8lr9d\" (UID: \"ebe524c8-2574-47dd-864c-b558de801715\") " pod="openshift-monitoring/node-exporter-8lr9d"
Sep 09 20:53:54.240802 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:54.240756    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-84zfh"
Sep 09 20:53:54.518721 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:54.518697    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-657fd6c54-84zfh"]
Sep 09 20:53:54.520832 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:53:54.520811    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8750f8b1_ad24_45b8_b483_8bc96f4d6198.slice/crio-a8ab14b21ac0a0c66c7ddc9adbacf620dcac42cb3f7b2696952167739ed1a55a WatchSource:0}: Error finding container a8ab14b21ac0a0c66c7ddc9adbacf620dcac42cb3f7b2696952167739ed1a55a: Status 404 returned error can't find the container with id a8ab14b21ac0a0c66c7ddc9adbacf620dcac42cb3f7b2696952167739ed1a55a
Sep 09 20:53:54.665256 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:54.664844    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ebe524c8-2574-47dd-864c-b558de801715-node-exporter-tls\") pod \"node-exporter-8lr9d\" (UID: \"ebe524c8-2574-47dd-864c-b558de801715\") " pod="openshift-monitoring/node-exporter-8lr9d"
Sep 09 20:53:54.667458 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:54.667409    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ebe524c8-2574-47dd-864c-b558de801715-node-exporter-tls\") pod \"node-exporter-8lr9d\" (UID: \"ebe524c8-2574-47dd-864c-b558de801715\") " pod="openshift-monitoring/node-exporter-8lr9d"
Sep 09 20:53:54.699186 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:54.699156    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-84zfh" event={"ID":"8750f8b1-ad24-45b8-b483-8bc96f4d6198","Type":"ContainerStarted","Data":"b93fa9a4d77bd37b0dc13858f7566f9d28d7dc548840e604ca9a72316894bae2"}
Sep 09 20:53:54.699280 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:54.699215    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-84zfh" event={"ID":"8750f8b1-ad24-45b8-b483-8bc96f4d6198","Type":"ContainerStarted","Data":"8f6fd535bc3c4f3d72cd7c25b2d868c79e7b0c6e83d05b8a0d773f0cae0e80e5"}
Sep 09 20:53:54.699280 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:54.699232    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-84zfh" event={"ID":"8750f8b1-ad24-45b8-b483-8bc96f4d6198","Type":"ContainerStarted","Data":"a8ab14b21ac0a0c66c7ddc9adbacf620dcac42cb3f7b2696952167739ed1a55a"}
Sep 09 20:53:54.880482 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:54.880386    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 09 20:53:54.886706 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:54.886680    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:53:54.889300 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:54.889279    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-hh79h\""
Sep 09 20:53:54.889940 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:54.889923    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 09 20:53:54.890033 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:54.889965    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 09 20:53:54.890033 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:54.890011    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 09 20:53:54.890152 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:54.890042    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 09 20:53:54.890373 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:54.890355    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 09 20:53:54.890539 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:54.890524    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 09 20:53:54.890707 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:54.890689    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 09 20:53:54.890893 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:54.890875    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 09 20:53:54.891005 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:54.890927    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 09 20:53:54.898701 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:54.898685    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-8lr9d"
Sep 09 20:53:54.908587 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:54.908565    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 09 20:53:54.909037 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:53:54.908984    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebe524c8_2574_47dd_864c_b558de801715.slice/crio-d737ad50f5e72ed60f193dac7c4193df3abdb60d22061e74335c0174a72b5532 WatchSource:0}: Error finding container d737ad50f5e72ed60f193dac7c4193df3abdb60d22061e74335c0174a72b5532: Status 404 returned error can't find the container with id d737ad50f5e72ed60f193dac7c4193df3abdb60d22061e74335c0174a72b5532
Sep 09 20:53:54.966879 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:54.966851    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/635ba76a-39eb-4f14-8eb3-b737f7c41b06-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"635ba76a-39eb-4f14-8eb3-b737f7c41b06\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:53:54.967205 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:54.966882    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/635ba76a-39eb-4f14-8eb3-b737f7c41b06-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"635ba76a-39eb-4f14-8eb3-b737f7c41b06\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:53:54.967205 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:54.966900    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/635ba76a-39eb-4f14-8eb3-b737f7c41b06-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"635ba76a-39eb-4f14-8eb3-b737f7c41b06\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:53:54.967205 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:54.966923    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/635ba76a-39eb-4f14-8eb3-b737f7c41b06-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"635ba76a-39eb-4f14-8eb3-b737f7c41b06\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:53:54.967205 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:54.966943    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/635ba76a-39eb-4f14-8eb3-b737f7c41b06-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"635ba76a-39eb-4f14-8eb3-b737f7c41b06\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:53:54.967205 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:54.967021    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/635ba76a-39eb-4f14-8eb3-b737f7c41b06-config-volume\") pod \"alertmanager-main-0\" (UID: \"635ba76a-39eb-4f14-8eb3-b737f7c41b06\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:53:54.967205 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:54.967063    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/635ba76a-39eb-4f14-8eb3-b737f7c41b06-config-out\") pod \"alertmanager-main-0\" (UID: \"635ba76a-39eb-4f14-8eb3-b737f7c41b06\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:53:54.967205 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:54.967091    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/635ba76a-39eb-4f14-8eb3-b737f7c41b06-tls-assets\") pod \"alertmanager-main-0\" (UID: \"635ba76a-39eb-4f14-8eb3-b737f7c41b06\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:53:54.967205 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:54.967145    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/635ba76a-39eb-4f14-8eb3-b737f7c41b06-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"635ba76a-39eb-4f14-8eb3-b737f7c41b06\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:53:54.967517 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:54.967209    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/635ba76a-39eb-4f14-8eb3-b737f7c41b06-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"635ba76a-39eb-4f14-8eb3-b737f7c41b06\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:53:54.967517 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:54.967231    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/635ba76a-39eb-4f14-8eb3-b737f7c41b06-web-config\") pod \"alertmanager-main-0\" (UID: \"635ba76a-39eb-4f14-8eb3-b737f7c41b06\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:53:54.967517 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:54.967256    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vpjkx\" (UniqueName: \"kubernetes.io/projected/635ba76a-39eb-4f14-8eb3-b737f7c41b06-kube-api-access-vpjkx\") pod \"alertmanager-main-0\" (UID: \"635ba76a-39eb-4f14-8eb3-b737f7c41b06\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:53:54.967517 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:54.967273    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/635ba76a-39eb-4f14-8eb3-b737f7c41b06-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"635ba76a-39eb-4f14-8eb3-b737f7c41b06\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:53:55.068222 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:55.068192    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/635ba76a-39eb-4f14-8eb3-b737f7c41b06-tls-assets\") pod \"alertmanager-main-0\" (UID: \"635ba76a-39eb-4f14-8eb3-b737f7c41b06\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:53:55.068356 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:55.068231    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/635ba76a-39eb-4f14-8eb3-b737f7c41b06-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"635ba76a-39eb-4f14-8eb3-b737f7c41b06\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:53:55.068356 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:55.068264    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/635ba76a-39eb-4f14-8eb3-b737f7c41b06-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"635ba76a-39eb-4f14-8eb3-b737f7c41b06\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:53:55.068356 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:55.068291    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/635ba76a-39eb-4f14-8eb3-b737f7c41b06-web-config\") pod \"alertmanager-main-0\" (UID: \"635ba76a-39eb-4f14-8eb3-b737f7c41b06\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:53:55.068356 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:55.068322    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpjkx\" (UniqueName: \"kubernetes.io/projected/635ba76a-39eb-4f14-8eb3-b737f7c41b06-kube-api-access-vpjkx\") pod \"alertmanager-main-0\" (UID: \"635ba76a-39eb-4f14-8eb3-b737f7c41b06\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:53:55.068356 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:55.068348    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/635ba76a-39eb-4f14-8eb3-b737f7c41b06-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"635ba76a-39eb-4f14-8eb3-b737f7c41b06\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:53:55.068633 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:55.068378    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/635ba76a-39eb-4f14-8eb3-b737f7c41b06-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"635ba76a-39eb-4f14-8eb3-b737f7c41b06\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:53:55.068633 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:55.068407    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/635ba76a-39eb-4f14-8eb3-b737f7c41b06-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"635ba76a-39eb-4f14-8eb3-b737f7c41b06\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:53:55.068633 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:55.068431    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/635ba76a-39eb-4f14-8eb3-b737f7c41b06-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"635ba76a-39eb-4f14-8eb3-b737f7c41b06\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:53:55.068633 ip-10-0-138-224 kubenswrapper[2579]: E0909 20:53:55.068497    2579 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found
Sep 09 20:53:55.068633 ip-10-0-138-224 kubenswrapper[2579]: E0909 20:53:55.068570    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/635ba76a-39eb-4f14-8eb3-b737f7c41b06-secret-alertmanager-main-tls podName:635ba76a-39eb-4f14-8eb3-b737f7c41b06 nodeName:}" failed. No retries permitted until 2026-09-09 20:53:55.568548681 +0000 UTC m=+92.808642803 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/635ba76a-39eb-4f14-8eb3-b737f7c41b06-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "635ba76a-39eb-4f14-8eb3-b737f7c41b06") : secret "alertmanager-main-tls" not found
Sep 09 20:53:55.068919 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:55.068497    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/635ba76a-39eb-4f14-8eb3-b737f7c41b06-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"635ba76a-39eb-4f14-8eb3-b737f7c41b06\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:53:55.068919 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:55.068799    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/635ba76a-39eb-4f14-8eb3-b737f7c41b06-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"635ba76a-39eb-4f14-8eb3-b737f7c41b06\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:53:55.068919 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:55.068837    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/635ba76a-39eb-4f14-8eb3-b737f7c41b06-config-volume\") pod \"alertmanager-main-0\" (UID: \"635ba76a-39eb-4f14-8eb3-b737f7c41b06\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:53:55.068919 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:55.068874    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/635ba76a-39eb-4f14-8eb3-b737f7c41b06-config-out\") pod \"alertmanager-main-0\" (UID: \"635ba76a-39eb-4f14-8eb3-b737f7c41b06\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:53:55.070181 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:55.069634    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/635ba76a-39eb-4f14-8eb3-b737f7c41b06-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"635ba76a-39eb-4f14-8eb3-b737f7c41b06\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:53:55.070181 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:55.069835    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/635ba76a-39eb-4f14-8eb3-b737f7c41b06-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"635ba76a-39eb-4f14-8eb3-b737f7c41b06\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:53:55.070181 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:55.070104    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/635ba76a-39eb-4f14-8eb3-b737f7c41b06-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"635ba76a-39eb-4f14-8eb3-b737f7c41b06\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:53:55.071574 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:55.071554    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/635ba76a-39eb-4f14-8eb3-b737f7c41b06-config-out\") pod \"alertmanager-main-0\" (UID: \"635ba76a-39eb-4f14-8eb3-b737f7c41b06\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:53:55.071673 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:55.071576    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/635ba76a-39eb-4f14-8eb3-b737f7c41b06-tls-assets\") pod \"alertmanager-main-0\" (UID: \"635ba76a-39eb-4f14-8eb3-b737f7c41b06\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:53:55.071673 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:55.071607    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/635ba76a-39eb-4f14-8eb3-b737f7c41b06-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"635ba76a-39eb-4f14-8eb3-b737f7c41b06\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:53:55.071779 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:55.071759    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/635ba76a-39eb-4f14-8eb3-b737f7c41b06-web-config\") pod \"alertmanager-main-0\" (UID: \"635ba76a-39eb-4f14-8eb3-b737f7c41b06\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:53:55.072014 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:55.071994    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/635ba76a-39eb-4f14-8eb3-b737f7c41b06-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"635ba76a-39eb-4f14-8eb3-b737f7c41b06\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:53:55.072796 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:55.072779    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/635ba76a-39eb-4f14-8eb3-b737f7c41b06-config-volume\") pod \"alertmanager-main-0\" (UID: \"635ba76a-39eb-4f14-8eb3-b737f7c41b06\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:53:55.073002 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:55.072974    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/635ba76a-39eb-4f14-8eb3-b737f7c41b06-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"635ba76a-39eb-4f14-8eb3-b737f7c41b06\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:53:55.073192 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:55.073175    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/635ba76a-39eb-4f14-8eb3-b737f7c41b06-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"635ba76a-39eb-4f14-8eb3-b737f7c41b06\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:53:55.083069 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:55.083050    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vpjkx\" (UniqueName: \"kubernetes.io/projected/635ba76a-39eb-4f14-8eb3-b737f7c41b06-kube-api-access-vpjkx\") pod \"alertmanager-main-0\" (UID: \"635ba76a-39eb-4f14-8eb3-b737f7c41b06\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:53:55.572831 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:55.572794    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/635ba76a-39eb-4f14-8eb3-b737f7c41b06-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"635ba76a-39eb-4f14-8eb3-b737f7c41b06\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:53:55.575525 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:55.575502    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/635ba76a-39eb-4f14-8eb3-b737f7c41b06-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"635ba76a-39eb-4f14-8eb3-b737f7c41b06\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:53:55.705460 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:55.705410    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8lr9d" event={"ID":"ebe524c8-2574-47dd-864c-b558de801715","Type":"ContainerStarted","Data":"d737ad50f5e72ed60f193dac7c4193df3abdb60d22061e74335c0174a72b5532"}
Sep 09 20:53:55.796244 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:55.796209    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:53:56.237985 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:56.237918    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 09 20:53:56.240116 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:53:56.240093    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod635ba76a_39eb_4f14_8eb3_b737f7c41b06.slice/crio-4992849be9b78ceeeee2870ac35a84965c51726448de44a690257cbc14950da2 WatchSource:0}: Error finding container 4992849be9b78ceeeee2870ac35a84965c51726448de44a690257cbc14950da2: Status 404 returned error can't find the container with id 4992849be9b78ceeeee2870ac35a84965c51726448de44a690257cbc14950da2
Sep 09 20:53:56.710353 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:56.710314    2579 generic.go:356] "Generic (PLEG): container finished" podID="ebe524c8-2574-47dd-864c-b558de801715" containerID="e72181ca141b19793d2e969c885080784868c969542f5733c539fe449535d545" exitCode=0
Sep 09 20:53:56.710535 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:56.710395    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8lr9d" event={"ID":"ebe524c8-2574-47dd-864c-b558de801715","Type":"ContainerDied","Data":"e72181ca141b19793d2e969c885080784868c969542f5733c539fe449535d545"}
Sep 09 20:53:56.711735 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:56.711707    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"635ba76a-39eb-4f14-8eb3-b737f7c41b06","Type":"ContainerStarted","Data":"4992849be9b78ceeeee2870ac35a84965c51726448de44a690257cbc14950da2"}
Sep 09 20:53:56.713787 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:56.713765    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-84zfh" event={"ID":"8750f8b1-ad24-45b8-b483-8bc96f4d6198","Type":"ContainerStarted","Data":"596f2d45d9d9a857ce4a41d8b309af73dbdbad9d64722800fa3d0f7060f16ccf"}
Sep 09 20:53:56.753731 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:56.753693    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-84zfh" podStartSLOduration=2.429695563 podStartE2EDuration="3.753679615s" totalImagesPullingTime="1.323984052s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:53:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:53:56.753567385 +0000 UTC m=+93.993661516" watchObservedRunningTime="2026-09-09 20:53:56.753679615 +0000 UTC m=+93.993773733"
Sep 09 20:53:56.882611 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:56.882579    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-759d4cf69c-mm926"]
Sep 09 20:53:56.886227 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:56.886202    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-759d4cf69c-mm926"
Sep 09 20:53:56.889107 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:56.888988    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Sep 09 20:53:56.889107 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:56.888989    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-jd5mk\""
Sep 09 20:53:56.889107 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:56.888989    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Sep 09 20:53:56.889107 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:56.889006    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Sep 09 20:53:56.889386 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:56.889368    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Sep 09 20:53:56.889453 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:56.889403    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Sep 09 20:53:56.889504 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:56.889465    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-4qhpd43fc9jrp\""
Sep 09 20:53:56.901226 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:56.900866    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-759d4cf69c-mm926"]
Sep 09 20:53:56.985244 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:56.985164    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/fff59aa3-255e-4b80-94b7-2b823917c67f-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-759d4cf69c-mm926\" (UID: \"fff59aa3-255e-4b80-94b7-2b823917c67f\") " pod="openshift-monitoring/thanos-querier-759d4cf69c-mm926"
Sep 09 20:53:56.985244 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:56.985208    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fff59aa3-255e-4b80-94b7-2b823917c67f-metrics-client-ca\") pod \"thanos-querier-759d4cf69c-mm926\" (UID: \"fff59aa3-255e-4b80-94b7-2b823917c67f\") " pod="openshift-monitoring/thanos-querier-759d4cf69c-mm926"
Sep 09 20:53:56.985516 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:56.985285    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fff59aa3-255e-4b80-94b7-2b823917c67f-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-759d4cf69c-mm926\" (UID: \"fff59aa3-255e-4b80-94b7-2b823917c67f\") " pod="openshift-monitoring/thanos-querier-759d4cf69c-mm926"
Sep 09 20:53:56.985516 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:56.985336    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fff59aa3-255e-4b80-94b7-2b823917c67f-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-759d4cf69c-mm926\" (UID: \"fff59aa3-255e-4b80-94b7-2b823917c67f\") " pod="openshift-monitoring/thanos-querier-759d4cf69c-mm926"
Sep 09 20:53:56.985516 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:56.985398    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/fff59aa3-255e-4b80-94b7-2b823917c67f-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-759d4cf69c-mm926\" (UID: \"fff59aa3-255e-4b80-94b7-2b823917c67f\") " pod="openshift-monitoring/thanos-querier-759d4cf69c-mm926"
Sep 09 20:53:56.985516 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:56.985484    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/fff59aa3-255e-4b80-94b7-2b823917c67f-secret-thanos-querier-tls\") pod \"thanos-querier-759d4cf69c-mm926\" (UID: \"fff59aa3-255e-4b80-94b7-2b823917c67f\") " pod="openshift-monitoring/thanos-querier-759d4cf69c-mm926"
Sep 09 20:53:56.985722 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:56.985552    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fff59aa3-255e-4b80-94b7-2b823917c67f-secret-grpc-tls\") pod \"thanos-querier-759d4cf69c-mm926\" (UID: \"fff59aa3-255e-4b80-94b7-2b823917c67f\") " pod="openshift-monitoring/thanos-querier-759d4cf69c-mm926"
Sep 09 20:53:56.985722 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:56.985586    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sng55\" (UniqueName: \"kubernetes.io/projected/fff59aa3-255e-4b80-94b7-2b823917c67f-kube-api-access-sng55\") pod \"thanos-querier-759d4cf69c-mm926\" (UID: \"fff59aa3-255e-4b80-94b7-2b823917c67f\") " pod="openshift-monitoring/thanos-querier-759d4cf69c-mm926"
Sep 09 20:53:57.086329 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:57.086291    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sng55\" (UniqueName: \"kubernetes.io/projected/fff59aa3-255e-4b80-94b7-2b823917c67f-kube-api-access-sng55\") pod \"thanos-querier-759d4cf69c-mm926\" (UID: \"fff59aa3-255e-4b80-94b7-2b823917c67f\") " pod="openshift-monitoring/thanos-querier-759d4cf69c-mm926"
Sep 09 20:53:57.086497 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:57.086345    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/fff59aa3-255e-4b80-94b7-2b823917c67f-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-759d4cf69c-mm926\" (UID: \"fff59aa3-255e-4b80-94b7-2b823917c67f\") " pod="openshift-monitoring/thanos-querier-759d4cf69c-mm926"
Sep 09 20:53:57.086497 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:57.086369    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fff59aa3-255e-4b80-94b7-2b823917c67f-metrics-client-ca\") pod \"thanos-querier-759d4cf69c-mm926\" (UID: \"fff59aa3-255e-4b80-94b7-2b823917c67f\") " pod="openshift-monitoring/thanos-querier-759d4cf69c-mm926"
Sep 09 20:53:57.086497 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:57.086406    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fff59aa3-255e-4b80-94b7-2b823917c67f-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-759d4cf69c-mm926\" (UID: \"fff59aa3-255e-4b80-94b7-2b823917c67f\") " pod="openshift-monitoring/thanos-querier-759d4cf69c-mm926"
Sep 09 20:53:57.086497 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:57.086464    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fff59aa3-255e-4b80-94b7-2b823917c67f-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-759d4cf69c-mm926\" (UID: \"fff59aa3-255e-4b80-94b7-2b823917c67f\") " pod="openshift-monitoring/thanos-querier-759d4cf69c-mm926"
Sep 09 20:53:57.086682 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:57.086511    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/fff59aa3-255e-4b80-94b7-2b823917c67f-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-759d4cf69c-mm926\" (UID: \"fff59aa3-255e-4b80-94b7-2b823917c67f\") " pod="openshift-monitoring/thanos-querier-759d4cf69c-mm926"
Sep 09 20:53:57.086682 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:57.086544    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/fff59aa3-255e-4b80-94b7-2b823917c67f-secret-thanos-querier-tls\") pod \"thanos-querier-759d4cf69c-mm926\" (UID: \"fff59aa3-255e-4b80-94b7-2b823917c67f\") " pod="openshift-monitoring/thanos-querier-759d4cf69c-mm926"
Sep 09 20:53:57.086682 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:57.086593    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fff59aa3-255e-4b80-94b7-2b823917c67f-secret-grpc-tls\") pod \"thanos-querier-759d4cf69c-mm926\" (UID: \"fff59aa3-255e-4b80-94b7-2b823917c67f\") " pod="openshift-monitoring/thanos-querier-759d4cf69c-mm926"
Sep 09 20:53:57.087260 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:57.087195    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fff59aa3-255e-4b80-94b7-2b823917c67f-metrics-client-ca\") pod \"thanos-querier-759d4cf69c-mm926\" (UID: \"fff59aa3-255e-4b80-94b7-2b823917c67f\") " pod="openshift-monitoring/thanos-querier-759d4cf69c-mm926"
Sep 09 20:53:57.089593 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:57.089568    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/fff59aa3-255e-4b80-94b7-2b823917c67f-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-759d4cf69c-mm926\" (UID: \"fff59aa3-255e-4b80-94b7-2b823917c67f\") " pod="openshift-monitoring/thanos-querier-759d4cf69c-mm926"
Sep 09 20:53:57.089866 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:57.089788    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fff59aa3-255e-4b80-94b7-2b823917c67f-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-759d4cf69c-mm926\" (UID: \"fff59aa3-255e-4b80-94b7-2b823917c67f\") " pod="openshift-monitoring/thanos-querier-759d4cf69c-mm926"
Sep 09 20:53:57.089866 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:57.089795    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/fff59aa3-255e-4b80-94b7-2b823917c67f-secret-thanos-querier-tls\") pod \"thanos-querier-759d4cf69c-mm926\" (UID: \"fff59aa3-255e-4b80-94b7-2b823917c67f\") " pod="openshift-monitoring/thanos-querier-759d4cf69c-mm926"
Sep 09 20:53:57.090101 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:57.090062    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fff59aa3-255e-4b80-94b7-2b823917c67f-secret-grpc-tls\") pod \"thanos-querier-759d4cf69c-mm926\" (UID: \"fff59aa3-255e-4b80-94b7-2b823917c67f\") " pod="openshift-monitoring/thanos-querier-759d4cf69c-mm926"
Sep 09 20:53:57.090286 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:57.090265    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/fff59aa3-255e-4b80-94b7-2b823917c67f-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-759d4cf69c-mm926\" (UID: \"fff59aa3-255e-4b80-94b7-2b823917c67f\") " pod="openshift-monitoring/thanos-querier-759d4cf69c-mm926"
Sep 09 20:53:57.090706 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:57.090686    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fff59aa3-255e-4b80-94b7-2b823917c67f-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-759d4cf69c-mm926\" (UID: \"fff59aa3-255e-4b80-94b7-2b823917c67f\") " pod="openshift-monitoring/thanos-querier-759d4cf69c-mm926"
Sep 09 20:53:57.094799 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:57.094775    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sng55\" (UniqueName: \"kubernetes.io/projected/fff59aa3-255e-4b80-94b7-2b823917c67f-kube-api-access-sng55\") pod \"thanos-querier-759d4cf69c-mm926\" (UID: \"fff59aa3-255e-4b80-94b7-2b823917c67f\") " pod="openshift-monitoring/thanos-querier-759d4cf69c-mm926"
Sep 09 20:53:57.198718 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:57.198691    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-759d4cf69c-mm926"
Sep 09 20:53:57.568403 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:57.568347    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-759d4cf69c-mm926"]
Sep 09 20:53:57.570310 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:53:57.570289    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfff59aa3_255e_4b80_94b7_2b823917c67f.slice/crio-d66eb247faf85e2e2f3e6c94f1624434aafc346f535f480a257f0aafd69aac6c WatchSource:0}: Error finding container d66eb247faf85e2e2f3e6c94f1624434aafc346f535f480a257f0aafd69aac6c: Status 404 returned error can't find the container with id d66eb247faf85e2e2f3e6c94f1624434aafc346f535f480a257f0aafd69aac6c
Sep 09 20:53:57.718259 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:57.718236    2579 generic.go:356] "Generic (PLEG): container finished" podID="635ba76a-39eb-4f14-8eb3-b737f7c41b06" containerID="1b927781582bae493a70e2cfae0621b890de2ab8e093309a533ba46fc71098e7" exitCode=0
Sep 09 20:53:57.718360 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:57.718326    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"635ba76a-39eb-4f14-8eb3-b737f7c41b06","Type":"ContainerDied","Data":"1b927781582bae493a70e2cfae0621b890de2ab8e093309a533ba46fc71098e7"}
Sep 09 20:53:57.720813 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:57.720773    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8lr9d" event={"ID":"ebe524c8-2574-47dd-864c-b558de801715","Type":"ContainerStarted","Data":"8e72a7a5502c46ed4853b92fcecd42f040450a5d7b69530b3e0ee7f136c0608c"}
Sep 09 20:53:57.720908 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:57.720840    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8lr9d" event={"ID":"ebe524c8-2574-47dd-864c-b558de801715","Type":"ContainerStarted","Data":"33b430144c0ea6af2e58a5bb9bbd7940d18cad37a5b9a12dfb5f999acf0a3636"}
Sep 09 20:53:57.722181 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:57.722159    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-759d4cf69c-mm926" event={"ID":"fff59aa3-255e-4b80-94b7-2b823917c67f","Type":"ContainerStarted","Data":"d66eb247faf85e2e2f3e6c94f1624434aafc346f535f480a257f0aafd69aac6c"}
Sep 09 20:53:57.794824 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:57.794774    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-8lr9d" podStartSLOduration=3.672522153 podStartE2EDuration="4.794761832s" totalImagesPullingTime="1.05130589s" totalInitContainerRuntime="70.933789ms" isStatefulPod=true podCreationTimestamp="2026-09-09 20:53:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:53:57.793263402 +0000 UTC m=+95.033357558" watchObservedRunningTime="2026-09-09 20:53:57.794761832 +0000 UTC m=+95.034855949"
Sep 09 20:53:58.668878 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:58.668842    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5495fd7677-lg774"]
Sep 09 20:53:58.672297 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:58.672271    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5495fd7677-lg774"
Sep 09 20:53:58.705218 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:58.705182    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5495fd7677-lg774"]
Sep 09 20:53:58.802517 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:58.802488    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pcddq\" (UniqueName: \"kubernetes.io/projected/83dac517-6dc0-4404-b8e5-77b11d91cd9a-kube-api-access-pcddq\") pod \"console-5495fd7677-lg774\" (UID: \"83dac517-6dc0-4404-b8e5-77b11d91cd9a\") " pod="openshift-console/console-5495fd7677-lg774"
Sep 09 20:53:58.802697 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:58.802565    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/83dac517-6dc0-4404-b8e5-77b11d91cd9a-console-oauth-config\") pod \"console-5495fd7677-lg774\" (UID: \"83dac517-6dc0-4404-b8e5-77b11d91cd9a\") " pod="openshift-console/console-5495fd7677-lg774"
Sep 09 20:53:58.802697 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:58.802604    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/83dac517-6dc0-4404-b8e5-77b11d91cd9a-oauth-serving-cert\") pod \"console-5495fd7677-lg774\" (UID: \"83dac517-6dc0-4404-b8e5-77b11d91cd9a\") " pod="openshift-console/console-5495fd7677-lg774"
Sep 09 20:53:58.802852 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:58.802817    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/83dac517-6dc0-4404-b8e5-77b11d91cd9a-service-ca\") pod \"console-5495fd7677-lg774\" (UID: \"83dac517-6dc0-4404-b8e5-77b11d91cd9a\") " pod="openshift-console/console-5495fd7677-lg774"
Sep 09 20:53:58.802974 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:58.802877    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/83dac517-6dc0-4404-b8e5-77b11d91cd9a-console-serving-cert\") pod \"console-5495fd7677-lg774\" (UID: \"83dac517-6dc0-4404-b8e5-77b11d91cd9a\") " pod="openshift-console/console-5495fd7677-lg774"
Sep 09 20:53:58.802974 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:58.802913    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/83dac517-6dc0-4404-b8e5-77b11d91cd9a-trusted-ca-bundle\") pod \"console-5495fd7677-lg774\" (UID: \"83dac517-6dc0-4404-b8e5-77b11d91cd9a\") " pod="openshift-console/console-5495fd7677-lg774"
Sep 09 20:53:58.802974 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:58.802944    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/83dac517-6dc0-4404-b8e5-77b11d91cd9a-console-config\") pod \"console-5495fd7677-lg774\" (UID: \"83dac517-6dc0-4404-b8e5-77b11d91cd9a\") " pod="openshift-console/console-5495fd7677-lg774"
Sep 09 20:53:58.903412 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:58.903381    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/83dac517-6dc0-4404-b8e5-77b11d91cd9a-console-oauth-config\") pod \"console-5495fd7677-lg774\" (UID: \"83dac517-6dc0-4404-b8e5-77b11d91cd9a\") " pod="openshift-console/console-5495fd7677-lg774"
Sep 09 20:53:58.903569 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:58.903421    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/83dac517-6dc0-4404-b8e5-77b11d91cd9a-oauth-serving-cert\") pod \"console-5495fd7677-lg774\" (UID: \"83dac517-6dc0-4404-b8e5-77b11d91cd9a\") " pod="openshift-console/console-5495fd7677-lg774"
Sep 09 20:53:58.903569 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:58.903477    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/83dac517-6dc0-4404-b8e5-77b11d91cd9a-service-ca\") pod \"console-5495fd7677-lg774\" (UID: \"83dac517-6dc0-4404-b8e5-77b11d91cd9a\") " pod="openshift-console/console-5495fd7677-lg774"
Sep 09 20:53:58.903671 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:58.903654    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/83dac517-6dc0-4404-b8e5-77b11d91cd9a-console-serving-cert\") pod \"console-5495fd7677-lg774\" (UID: \"83dac517-6dc0-4404-b8e5-77b11d91cd9a\") " pod="openshift-console/console-5495fd7677-lg774"
Sep 09 20:53:58.903788 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:58.903706    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/83dac517-6dc0-4404-b8e5-77b11d91cd9a-trusted-ca-bundle\") pod \"console-5495fd7677-lg774\" (UID: \"83dac517-6dc0-4404-b8e5-77b11d91cd9a\") " pod="openshift-console/console-5495fd7677-lg774"
Sep 09 20:53:58.903788 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:58.903738    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/83dac517-6dc0-4404-b8e5-77b11d91cd9a-console-config\") pod \"console-5495fd7677-lg774\" (UID: \"83dac517-6dc0-4404-b8e5-77b11d91cd9a\") " pod="openshift-console/console-5495fd7677-lg774"
Sep 09 20:53:58.903788 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:58.903773    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pcddq\" (UniqueName: \"kubernetes.io/projected/83dac517-6dc0-4404-b8e5-77b11d91cd9a-kube-api-access-pcddq\") pod \"console-5495fd7677-lg774\" (UID: \"83dac517-6dc0-4404-b8e5-77b11d91cd9a\") " pod="openshift-console/console-5495fd7677-lg774"
Sep 09 20:53:58.904454 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:58.904227    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/83dac517-6dc0-4404-b8e5-77b11d91cd9a-oauth-serving-cert\") pod \"console-5495fd7677-lg774\" (UID: \"83dac517-6dc0-4404-b8e5-77b11d91cd9a\") " pod="openshift-console/console-5495fd7677-lg774"
Sep 09 20:53:58.904454 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:58.904239    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/83dac517-6dc0-4404-b8e5-77b11d91cd9a-service-ca\") pod \"console-5495fd7677-lg774\" (UID: \"83dac517-6dc0-4404-b8e5-77b11d91cd9a\") " pod="openshift-console/console-5495fd7677-lg774"
Sep 09 20:53:58.904653 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:58.904557    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/83dac517-6dc0-4404-b8e5-77b11d91cd9a-console-config\") pod \"console-5495fd7677-lg774\" (UID: \"83dac517-6dc0-4404-b8e5-77b11d91cd9a\") " pod="openshift-console/console-5495fd7677-lg774"
Sep 09 20:53:58.904714 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:58.904646    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/83dac517-6dc0-4404-b8e5-77b11d91cd9a-trusted-ca-bundle\") pod \"console-5495fd7677-lg774\" (UID: \"83dac517-6dc0-4404-b8e5-77b11d91cd9a\") " pod="openshift-console/console-5495fd7677-lg774"
Sep 09 20:53:58.906390 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:58.906346    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/83dac517-6dc0-4404-b8e5-77b11d91cd9a-console-oauth-config\") pod \"console-5495fd7677-lg774\" (UID: \"83dac517-6dc0-4404-b8e5-77b11d91cd9a\") " pod="openshift-console/console-5495fd7677-lg774"
Sep 09 20:53:58.906867 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:58.906849    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/83dac517-6dc0-4404-b8e5-77b11d91cd9a-console-serving-cert\") pod \"console-5495fd7677-lg774\" (UID: \"83dac517-6dc0-4404-b8e5-77b11d91cd9a\") " pod="openshift-console/console-5495fd7677-lg774"
Sep 09 20:53:58.918222 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:58.918198    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pcddq\" (UniqueName: \"kubernetes.io/projected/83dac517-6dc0-4404-b8e5-77b11d91cd9a-kube-api-access-pcddq\") pod \"console-5495fd7677-lg774\" (UID: \"83dac517-6dc0-4404-b8e5-77b11d91cd9a\") " pod="openshift-console/console-5495fd7677-lg774"
Sep 09 20:53:58.983112 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:58.983050    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5495fd7677-lg774"
Sep 09 20:53:59.732692 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:59.732569    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-759d4cf69c-mm926" event={"ID":"fff59aa3-255e-4b80-94b7-2b823917c67f","Type":"ContainerStarted","Data":"a2ed7efcb31c58d4c4cedc1c3179988205373c663069629c886f36bc743430b4"}
Sep 09 20:53:59.732692 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:59.732662    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-759d4cf69c-mm926" event={"ID":"fff59aa3-255e-4b80-94b7-2b823917c67f","Type":"ContainerStarted","Data":"c9610e0a73cb5fbc3946d7bbb5bcdcfc52ee04c708aa586fc28f3e9558e60c83"}
Sep 09 20:53:59.735140 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:59.735114    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"635ba76a-39eb-4f14-8eb3-b737f7c41b06","Type":"ContainerStarted","Data":"dd8707e0b982d2b9518bc1e14edc4d96553cffd9f48ed5c403dc5d81f5e9e3eb"}
Sep 09 20:53:59.735225 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:59.735181    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"635ba76a-39eb-4f14-8eb3-b737f7c41b06","Type":"ContainerStarted","Data":"c7128cb2bbf82c3a0d9138b55dbc6c7d138ba13d48f0f7a77ead363c8ac4fdca"}
Sep 09 20:53:59.826595 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:53:59.826568    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5495fd7677-lg774"]
Sep 09 20:53:59.834706 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:53:59.834672    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83dac517_6dc0_4404_b8e5_77b11d91cd9a.slice/crio-e6924229be342b678c6e4bba52ad3988d53f8c176cb12a82151e6d88b4d3a37e WatchSource:0}: Error finding container e6924229be342b678c6e4bba52ad3988d53f8c176cb12a82151e6d88b4d3a37e: Status 404 returned error can't find the container with id e6924229be342b678c6e4bba52ad3988d53f8c176cb12a82151e6d88b4d3a37e
Sep 09 20:54:00.742877 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:54:00.742837    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"635ba76a-39eb-4f14-8eb3-b737f7c41b06","Type":"ContainerStarted","Data":"60ad2131ec3eb6ea5cb3b6b301f3961c054cda597093e6eb87c318a74f47f4ca"}
Sep 09 20:54:00.743211 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:54:00.742921    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"635ba76a-39eb-4f14-8eb3-b737f7c41b06","Type":"ContainerStarted","Data":"b35ffad7008a0f1e0f496ee14fc6458285d9773a924d3ec291721325936338a2"}
Sep 09 20:54:00.743211 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:54:00.742976    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"635ba76a-39eb-4f14-8eb3-b737f7c41b06","Type":"ContainerStarted","Data":"ff902667b182e31061cbf572c763a246540db1f251944c72bc52f21fa2a4de7e"}
Sep 09 20:54:00.743211 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:54:00.743019    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"635ba76a-39eb-4f14-8eb3-b737f7c41b06","Type":"ContainerStarted","Data":"87ad037a5d558c7a2bad84cc632b6882d5ffa038212f8ce068868d25fa38fa20"}
Sep 09 20:54:00.744647 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:54:00.744622    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5495fd7677-lg774" event={"ID":"83dac517-6dc0-4404-b8e5-77b11d91cd9a","Type":"ContainerStarted","Data":"2b9291d518c957dc3e5384cd8720fd8c6f394587b52e5bf826ecdd9bc6a978d9"}
Sep 09 20:54:00.744743 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:54:00.744665    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5495fd7677-lg774" event={"ID":"83dac517-6dc0-4404-b8e5-77b11d91cd9a","Type":"ContainerStarted","Data":"e6924229be342b678c6e4bba52ad3988d53f8c176cb12a82151e6d88b4d3a37e"}
Sep 09 20:54:00.747455 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:54:00.747421    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-759d4cf69c-mm926" event={"ID":"fff59aa3-255e-4b80-94b7-2b823917c67f","Type":"ContainerStarted","Data":"2dad83dfd616b9358904807cf577abbcd38779d350889137b61196b78c800f15"}
Sep 09 20:54:00.747534 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:54:00.747496    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-759d4cf69c-mm926" event={"ID":"fff59aa3-255e-4b80-94b7-2b823917c67f","Type":"ContainerStarted","Data":"ee46c79cee50e79f54596a26c2a92f8639c4a36b421664d659974cece8fc1332"}
Sep 09 20:54:00.783214 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:54:00.783171    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.073878385 podStartE2EDuration="6.783156351s" totalImagesPullingTime="3.694038731s" totalInitContainerRuntime="15.239235ms" isStatefulPod=false podCreationTimestamp="2026-09-09 20:53:54 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:54:00.783047124 +0000 UTC m=+98.023141241" watchObservedRunningTime="2026-09-09 20:54:00.783156351 +0000 UTC m=+98.023250471"
Sep 09 20:54:00.805517 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:54:00.805475    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5495fd7677-lg774" podStartSLOduration=2.805433176 podStartE2EDuration="2.805433176s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:53:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:54:00.804553423 +0000 UTC m=+98.044647542" watchObservedRunningTime="2026-09-09 20:54:00.805433176 +0000 UTC m=+98.045527293"
Sep 09 20:54:01.754958 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:54:01.754917    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-759d4cf69c-mm926" event={"ID":"fff59aa3-255e-4b80-94b7-2b823917c67f","Type":"ContainerStarted","Data":"24de50984bf2fef13a394cff648501ea4a558dd30a0f89520dd1737ca0211ce9"}
Sep 09 20:54:01.755359 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:54:01.755046    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-759d4cf69c-mm926" event={"ID":"fff59aa3-255e-4b80-94b7-2b823917c67f","Type":"ContainerStarted","Data":"ead2184b77efed5720df28c8c8a6123ba9aa8fef920663259b95fd2988a2b1be"}
Sep 09 20:54:01.755359 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:54:01.755101    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-759d4cf69c-mm926"
Sep 09 20:54:01.778794 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:54:01.778744    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-759d4cf69c-mm926" podStartSLOduration=2.947571983 podStartE2EDuration="5.778729186s" totalImagesPullingTime="2.831157203s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:53:56 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:54:01.776367015 +0000 UTC m=+99.016461134" watchObservedRunningTime="2026-09-09 20:54:01.778729186 +0000 UTC m=+99.018823305"
Sep 09 20:54:06.630346 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:54:06.630309    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-b6qr7"
Sep 09 20:54:07.765831 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:54:07.765805    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-759d4cf69c-mm926"
Sep 09 20:54:08.984239 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:54:08.984201    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5495fd7677-lg774"
Sep 09 20:54:08.984643 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:54:08.984268    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5495fd7677-lg774"
Sep 09 20:54:08.988937 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:54:08.988910    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5495fd7677-lg774"
Sep 09 20:54:09.792287 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:54:09.792260    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5495fd7677-lg774"
Sep 09 20:54:09.840654 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:54:09.840624    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-655476f677-9tncl"]
Sep 09 20:54:34.860517 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:54:34.860431    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-655476f677-9tncl" podUID="f4343b3e-2a6c-41f8-95df-f9a50bcc3d8d" containerName="console" containerID="cri-o://dfe84bc04576025a0f854e68a34f3207533f4b1f5fd2f7284f926443889a2b2b" gracePeriod=15
Sep 09 20:54:35.102357 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:54:35.102333    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-655476f677-9tncl"
Sep 09 20:54:35.198594 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:54:35.198551    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f4343b3e-2a6c-41f8-95df-f9a50bcc3d8d-trusted-ca-bundle\") pod \"f4343b3e-2a6c-41f8-95df-f9a50bcc3d8d\" (UID: \"f4343b3e-2a6c-41f8-95df-f9a50bcc3d8d\") "
Sep 09 20:54:35.198757 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:54:35.198609    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f4343b3e-2a6c-41f8-95df-f9a50bcc3d8d-oauth-serving-cert\") pod \"f4343b3e-2a6c-41f8-95df-f9a50bcc3d8d\" (UID: \"f4343b3e-2a6c-41f8-95df-f9a50bcc3d8d\") "
Sep 09 20:54:35.198757 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:54:35.198631    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f4343b3e-2a6c-41f8-95df-f9a50bcc3d8d-console-serving-cert\") pod \"f4343b3e-2a6c-41f8-95df-f9a50bcc3d8d\" (UID: \"f4343b3e-2a6c-41f8-95df-f9a50bcc3d8d\") "
Sep 09 20:54:35.198757 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:54:35.198663    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f4343b3e-2a6c-41f8-95df-f9a50bcc3d8d-console-oauth-config\") pod \"f4343b3e-2a6c-41f8-95df-f9a50bcc3d8d\" (UID: \"f4343b3e-2a6c-41f8-95df-f9a50bcc3d8d\") "
Sep 09 20:54:35.198757 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:54:35.198688    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f4343b3e-2a6c-41f8-95df-f9a50bcc3d8d-service-ca\") pod \"f4343b3e-2a6c-41f8-95df-f9a50bcc3d8d\" (UID: \"f4343b3e-2a6c-41f8-95df-f9a50bcc3d8d\") "
Sep 09 20:54:35.198757 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:54:35.198715    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f66ww\" (UniqueName: \"kubernetes.io/projected/f4343b3e-2a6c-41f8-95df-f9a50bcc3d8d-kube-api-access-f66ww\") pod \"f4343b3e-2a6c-41f8-95df-f9a50bcc3d8d\" (UID: \"f4343b3e-2a6c-41f8-95df-f9a50bcc3d8d\") "
Sep 09 20:54:35.199009 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:54:35.198772    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f4343b3e-2a6c-41f8-95df-f9a50bcc3d8d-console-config\") pod \"f4343b3e-2a6c-41f8-95df-f9a50bcc3d8d\" (UID: \"f4343b3e-2a6c-41f8-95df-f9a50bcc3d8d\") "
Sep 09 20:54:35.199173 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:54:35.199136    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4343b3e-2a6c-41f8-95df-f9a50bcc3d8d-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "f4343b3e-2a6c-41f8-95df-f9a50bcc3d8d" (UID: "f4343b3e-2a6c-41f8-95df-f9a50bcc3d8d"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:54:35.199173 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:54:35.199158    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4343b3e-2a6c-41f8-95df-f9a50bcc3d8d-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "f4343b3e-2a6c-41f8-95df-f9a50bcc3d8d" (UID: "f4343b3e-2a6c-41f8-95df-f9a50bcc3d8d"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:54:35.199357 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:54:35.199170    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4343b3e-2a6c-41f8-95df-f9a50bcc3d8d-service-ca" (OuterVolumeSpecName: "service-ca") pod "f4343b3e-2a6c-41f8-95df-f9a50bcc3d8d" (UID: "f4343b3e-2a6c-41f8-95df-f9a50bcc3d8d"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:54:35.199357 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:54:35.199269    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4343b3e-2a6c-41f8-95df-f9a50bcc3d8d-console-config" (OuterVolumeSpecName: "console-config") pod "f4343b3e-2a6c-41f8-95df-f9a50bcc3d8d" (UID: "f4343b3e-2a6c-41f8-95df-f9a50bcc3d8d"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:54:35.201222 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:54:35.201194    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f4343b3e-2a6c-41f8-95df-f9a50bcc3d8d-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "f4343b3e-2a6c-41f8-95df-f9a50bcc3d8d" (UID: "f4343b3e-2a6c-41f8-95df-f9a50bcc3d8d"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:54:35.201222 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:54:35.201209    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f4343b3e-2a6c-41f8-95df-f9a50bcc3d8d-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "f4343b3e-2a6c-41f8-95df-f9a50bcc3d8d" (UID: "f4343b3e-2a6c-41f8-95df-f9a50bcc3d8d"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:54:35.201222 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:54:35.201190    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4343b3e-2a6c-41f8-95df-f9a50bcc3d8d-kube-api-access-f66ww" (OuterVolumeSpecName: "kube-api-access-f66ww") pod "f4343b3e-2a6c-41f8-95df-f9a50bcc3d8d" (UID: "f4343b3e-2a6c-41f8-95df-f9a50bcc3d8d"). InnerVolumeSpecName "kube-api-access-f66ww". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:54:35.299993 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:54:35.299972    2579 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f4343b3e-2a6c-41f8-95df-f9a50bcc3d8d-console-config\") on node \"ip-10-0-138-224.ec2.internal\" DevicePath \"\""
Sep 09 20:54:35.299993 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:54:35.299994    2579 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f4343b3e-2a6c-41f8-95df-f9a50bcc3d8d-trusted-ca-bundle\") on node \"ip-10-0-138-224.ec2.internal\" DevicePath \"\""
Sep 09 20:54:35.300126 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:54:35.300003    2579 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f4343b3e-2a6c-41f8-95df-f9a50bcc3d8d-oauth-serving-cert\") on node \"ip-10-0-138-224.ec2.internal\" DevicePath \"\""
Sep 09 20:54:35.300126 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:54:35.300013    2579 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f4343b3e-2a6c-41f8-95df-f9a50bcc3d8d-console-serving-cert\") on node \"ip-10-0-138-224.ec2.internal\" DevicePath \"\""
Sep 09 20:54:35.300126 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:54:35.300022    2579 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f4343b3e-2a6c-41f8-95df-f9a50bcc3d8d-console-oauth-config\") on node \"ip-10-0-138-224.ec2.internal\" DevicePath \"\""
Sep 09 20:54:35.300126 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:54:35.300030    2579 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f4343b3e-2a6c-41f8-95df-f9a50bcc3d8d-service-ca\") on node \"ip-10-0-138-224.ec2.internal\" DevicePath \"\""
Sep 09 20:54:35.300126 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:54:35.300039    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f66ww\" (UniqueName: \"kubernetes.io/projected/f4343b3e-2a6c-41f8-95df-f9a50bcc3d8d-kube-api-access-f66ww\") on node \"ip-10-0-138-224.ec2.internal\" DevicePath \"\""
Sep 09 20:54:35.885749 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:54:35.885715    2579 generic.go:356] "Generic (PLEG): container finished" podID="f4343b3e-2a6c-41f8-95df-f9a50bcc3d8d" containerID="dfe84bc04576025a0f854e68a34f3207533f4b1f5fd2f7284f926443889a2b2b" exitCode=0
Sep 09 20:54:35.886146 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:54:35.885756    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-655476f677-9tncl"
Sep 09 20:54:35.886146 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:54:35.885801    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-655476f677-9tncl" event={"ID":"f4343b3e-2a6c-41f8-95df-f9a50bcc3d8d","Type":"ContainerDied","Data":"dfe84bc04576025a0f854e68a34f3207533f4b1f5fd2f7284f926443889a2b2b"}
Sep 09 20:54:35.886146 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:54:35.885855    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-655476f677-9tncl" event={"ID":"f4343b3e-2a6c-41f8-95df-f9a50bcc3d8d","Type":"ContainerDied","Data":"a0a630da8c04f76fc5da9fd061388a641b7d07248898782e360e81803b123b7e"}
Sep 09 20:54:35.886146 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:54:35.885884    2579 scope.go:117] "RemoveContainer" containerID="dfe84bc04576025a0f854e68a34f3207533f4b1f5fd2f7284f926443889a2b2b"
Sep 09 20:54:35.894141 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:54:35.894121    2579 scope.go:117] "RemoveContainer" containerID="dfe84bc04576025a0f854e68a34f3207533f4b1f5fd2f7284f926443889a2b2b"
Sep 09 20:54:35.894415 ip-10-0-138-224 kubenswrapper[2579]: E0909 20:54:35.894396    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dfe84bc04576025a0f854e68a34f3207533f4b1f5fd2f7284f926443889a2b2b\": container with ID starting with dfe84bc04576025a0f854e68a34f3207533f4b1f5fd2f7284f926443889a2b2b not found: ID does not exist" containerID="dfe84bc04576025a0f854e68a34f3207533f4b1f5fd2f7284f926443889a2b2b"
Sep 09 20:54:35.894475 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:54:35.894424    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dfe84bc04576025a0f854e68a34f3207533f4b1f5fd2f7284f926443889a2b2b"} err="failed to get container status \"dfe84bc04576025a0f854e68a34f3207533f4b1f5fd2f7284f926443889a2b2b\": rpc error: code = NotFound desc = could not find container \"dfe84bc04576025a0f854e68a34f3207533f4b1f5fd2f7284f926443889a2b2b\": container with ID starting with dfe84bc04576025a0f854e68a34f3207533f4b1f5fd2f7284f926443889a2b2b not found: ID does not exist"
Sep 09 20:54:35.903026 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:54:35.903001    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-655476f677-9tncl"]
Sep 09 20:54:35.906402 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:54:35.906376    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-655476f677-9tncl"]
Sep 09 20:54:37.290256 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:54:37.290216    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f4343b3e-2a6c-41f8-95df-f9a50bcc3d8d" path="/var/lib/kubelet/pods/f4343b3e-2a6c-41f8-95df-f9a50bcc3d8d/volumes"
Sep 09 20:55:14.150004 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:14.149969    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 09 20:55:14.150504 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:14.150265    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="635ba76a-39eb-4f14-8eb3-b737f7c41b06" containerName="alertmanager" containerID="cri-o://c7128cb2bbf82c3a0d9138b55dbc6c7d138ba13d48f0f7a77ead363c8ac4fdca" gracePeriod=120
Sep 09 20:55:14.150504 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:14.150304    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="635ba76a-39eb-4f14-8eb3-b737f7c41b06" containerName="kube-rbac-proxy-metric" containerID="cri-o://b35ffad7008a0f1e0f496ee14fc6458285d9773a924d3ec291721325936338a2" gracePeriod=120
Sep 09 20:55:14.150504 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:14.150356    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="635ba76a-39eb-4f14-8eb3-b737f7c41b06" containerName="kube-rbac-proxy-web" containerID="cri-o://87ad037a5d558c7a2bad84cc632b6882d5ffa038212f8ce068868d25fa38fa20" gracePeriod=120
Sep 09 20:55:14.150504 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:14.150406    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="635ba76a-39eb-4f14-8eb3-b737f7c41b06" containerName="config-reloader" containerID="cri-o://dd8707e0b982d2b9518bc1e14edc4d96553cffd9f48ed5c403dc5d81f5e9e3eb" gracePeriod=120
Sep 09 20:55:14.150504 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:14.150390    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="635ba76a-39eb-4f14-8eb3-b737f7c41b06" containerName="kube-rbac-proxy" containerID="cri-o://ff902667b182e31061cbf572c763a246540db1f251944c72bc52f21fa2a4de7e" gracePeriod=120
Sep 09 20:55:14.150504 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:14.150388    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="635ba76a-39eb-4f14-8eb3-b737f7c41b06" containerName="prom-label-proxy" containerID="cri-o://60ad2131ec3eb6ea5cb3b6b301f3961c054cda597093e6eb87c318a74f47f4ca" gracePeriod=120
Sep 09 20:55:15.045171 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:15.045135    2579 generic.go:356] "Generic (PLEG): container finished" podID="635ba76a-39eb-4f14-8eb3-b737f7c41b06" containerID="60ad2131ec3eb6ea5cb3b6b301f3961c054cda597093e6eb87c318a74f47f4ca" exitCode=0
Sep 09 20:55:15.045171 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:15.045160    2579 generic.go:356] "Generic (PLEG): container finished" podID="635ba76a-39eb-4f14-8eb3-b737f7c41b06" containerID="ff902667b182e31061cbf572c763a246540db1f251944c72bc52f21fa2a4de7e" exitCode=0
Sep 09 20:55:15.045171 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:15.045169    2579 generic.go:356] "Generic (PLEG): container finished" podID="635ba76a-39eb-4f14-8eb3-b737f7c41b06" containerID="dd8707e0b982d2b9518bc1e14edc4d96553cffd9f48ed5c403dc5d81f5e9e3eb" exitCode=0
Sep 09 20:55:15.045171 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:15.045178    2579 generic.go:356] "Generic (PLEG): container finished" podID="635ba76a-39eb-4f14-8eb3-b737f7c41b06" containerID="c7128cb2bbf82c3a0d9138b55dbc6c7d138ba13d48f0f7a77ead363c8ac4fdca" exitCode=0
Sep 09 20:55:15.045429 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:15.045201    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"635ba76a-39eb-4f14-8eb3-b737f7c41b06","Type":"ContainerDied","Data":"60ad2131ec3eb6ea5cb3b6b301f3961c054cda597093e6eb87c318a74f47f4ca"}
Sep 09 20:55:15.045429 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:15.045261    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"635ba76a-39eb-4f14-8eb3-b737f7c41b06","Type":"ContainerDied","Data":"ff902667b182e31061cbf572c763a246540db1f251944c72bc52f21fa2a4de7e"}
Sep 09 20:55:15.045429 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:15.045286    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"635ba76a-39eb-4f14-8eb3-b737f7c41b06","Type":"ContainerDied","Data":"dd8707e0b982d2b9518bc1e14edc4d96553cffd9f48ed5c403dc5d81f5e9e3eb"}
Sep 09 20:55:15.045429 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:15.045310    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"635ba76a-39eb-4f14-8eb3-b737f7c41b06","Type":"ContainerDied","Data":"c7128cb2bbf82c3a0d9138b55dbc6c7d138ba13d48f0f7a77ead363c8ac4fdca"}
Sep 09 20:55:15.393942 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:15.393921    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:55:15.515275 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:15.515247    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/635ba76a-39eb-4f14-8eb3-b737f7c41b06-secret-alertmanager-kube-rbac-proxy-metric\") pod \"635ba76a-39eb-4f14-8eb3-b737f7c41b06\" (UID: \"635ba76a-39eb-4f14-8eb3-b737f7c41b06\") "
Sep 09 20:55:15.515275 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:15.515278    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/635ba76a-39eb-4f14-8eb3-b737f7c41b06-metrics-client-ca\") pod \"635ba76a-39eb-4f14-8eb3-b737f7c41b06\" (UID: \"635ba76a-39eb-4f14-8eb3-b737f7c41b06\") "
Sep 09 20:55:15.515511 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:15.515308    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/635ba76a-39eb-4f14-8eb3-b737f7c41b06-cluster-tls-config\") pod \"635ba76a-39eb-4f14-8eb3-b737f7c41b06\" (UID: \"635ba76a-39eb-4f14-8eb3-b737f7c41b06\") "
Sep 09 20:55:15.515511 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:15.515333    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/635ba76a-39eb-4f14-8eb3-b737f7c41b06-web-config\") pod \"635ba76a-39eb-4f14-8eb3-b737f7c41b06\" (UID: \"635ba76a-39eb-4f14-8eb3-b737f7c41b06\") "
Sep 09 20:55:15.515511 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:15.515358    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/635ba76a-39eb-4f14-8eb3-b737f7c41b06-config-volume\") pod \"635ba76a-39eb-4f14-8eb3-b737f7c41b06\" (UID: \"635ba76a-39eb-4f14-8eb3-b737f7c41b06\") "
Sep 09 20:55:15.515511 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:15.515381    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/635ba76a-39eb-4f14-8eb3-b737f7c41b06-tls-assets\") pod \"635ba76a-39eb-4f14-8eb3-b737f7c41b06\" (UID: \"635ba76a-39eb-4f14-8eb3-b737f7c41b06\") "
Sep 09 20:55:15.515511 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:15.515461    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vpjkx\" (UniqueName: \"kubernetes.io/projected/635ba76a-39eb-4f14-8eb3-b737f7c41b06-kube-api-access-vpjkx\") pod \"635ba76a-39eb-4f14-8eb3-b737f7c41b06\" (UID: \"635ba76a-39eb-4f14-8eb3-b737f7c41b06\") "
Sep 09 20:55:15.515793 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:15.515522    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/635ba76a-39eb-4f14-8eb3-b737f7c41b06-secret-alertmanager-main-tls\") pod \"635ba76a-39eb-4f14-8eb3-b737f7c41b06\" (UID: \"635ba76a-39eb-4f14-8eb3-b737f7c41b06\") "
Sep 09 20:55:15.515793 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:15.515554    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/635ba76a-39eb-4f14-8eb3-b737f7c41b06-alertmanager-main-db\") pod \"635ba76a-39eb-4f14-8eb3-b737f7c41b06\" (UID: \"635ba76a-39eb-4f14-8eb3-b737f7c41b06\") "
Sep 09 20:55:15.515793 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:15.515612    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/635ba76a-39eb-4f14-8eb3-b737f7c41b06-secret-alertmanager-kube-rbac-proxy-web\") pod \"635ba76a-39eb-4f14-8eb3-b737f7c41b06\" (UID: \"635ba76a-39eb-4f14-8eb3-b737f7c41b06\") "
Sep 09 20:55:15.515793 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:15.515648    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/635ba76a-39eb-4f14-8eb3-b737f7c41b06-secret-alertmanager-kube-rbac-proxy\") pod \"635ba76a-39eb-4f14-8eb3-b737f7c41b06\" (UID: \"635ba76a-39eb-4f14-8eb3-b737f7c41b06\") "
Sep 09 20:55:15.515793 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:15.515675    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/635ba76a-39eb-4f14-8eb3-b737f7c41b06-config-out\") pod \"635ba76a-39eb-4f14-8eb3-b737f7c41b06\" (UID: \"635ba76a-39eb-4f14-8eb3-b737f7c41b06\") "
Sep 09 20:55:15.515793 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:15.515683    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/635ba76a-39eb-4f14-8eb3-b737f7c41b06-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "635ba76a-39eb-4f14-8eb3-b737f7c41b06" (UID: "635ba76a-39eb-4f14-8eb3-b737f7c41b06"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:55:15.515793 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:15.515704    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/635ba76a-39eb-4f14-8eb3-b737f7c41b06-alertmanager-trusted-ca-bundle\") pod \"635ba76a-39eb-4f14-8eb3-b737f7c41b06\" (UID: \"635ba76a-39eb-4f14-8eb3-b737f7c41b06\") "
Sep 09 20:55:15.516286 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:15.516153    2579 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/635ba76a-39eb-4f14-8eb3-b737f7c41b06-metrics-client-ca\") on node \"ip-10-0-138-224.ec2.internal\" DevicePath \"\""
Sep 09 20:55:15.516826 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:15.516793    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/635ba76a-39eb-4f14-8eb3-b737f7c41b06-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "635ba76a-39eb-4f14-8eb3-b737f7c41b06" (UID: "635ba76a-39eb-4f14-8eb3-b737f7c41b06"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:55:15.516942 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:15.516842    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/635ba76a-39eb-4f14-8eb3-b737f7c41b06-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "635ba76a-39eb-4f14-8eb3-b737f7c41b06" (UID: "635ba76a-39eb-4f14-8eb3-b737f7c41b06"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:55:15.518370 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:15.518342    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/635ba76a-39eb-4f14-8eb3-b737f7c41b06-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "635ba76a-39eb-4f14-8eb3-b737f7c41b06" (UID: "635ba76a-39eb-4f14-8eb3-b737f7c41b06"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:55:15.518974 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:15.518757    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/635ba76a-39eb-4f14-8eb3-b737f7c41b06-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "635ba76a-39eb-4f14-8eb3-b737f7c41b06" (UID: "635ba76a-39eb-4f14-8eb3-b737f7c41b06"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:55:15.519085 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:15.519040    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/635ba76a-39eb-4f14-8eb3-b737f7c41b06-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "635ba76a-39eb-4f14-8eb3-b737f7c41b06" (UID: "635ba76a-39eb-4f14-8eb3-b737f7c41b06"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:55:15.519148 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:15.519071    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/635ba76a-39eb-4f14-8eb3-b737f7c41b06-config-volume" (OuterVolumeSpecName: "config-volume") pod "635ba76a-39eb-4f14-8eb3-b737f7c41b06" (UID: "635ba76a-39eb-4f14-8eb3-b737f7c41b06"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:55:15.519630 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:15.519599    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/635ba76a-39eb-4f14-8eb3-b737f7c41b06-kube-api-access-vpjkx" (OuterVolumeSpecName: "kube-api-access-vpjkx") pod "635ba76a-39eb-4f14-8eb3-b737f7c41b06" (UID: "635ba76a-39eb-4f14-8eb3-b737f7c41b06"). InnerVolumeSpecName "kube-api-access-vpjkx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:55:15.519962 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:15.519936    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/635ba76a-39eb-4f14-8eb3-b737f7c41b06-config-out" (OuterVolumeSpecName: "config-out") pod "635ba76a-39eb-4f14-8eb3-b737f7c41b06" (UID: "635ba76a-39eb-4f14-8eb3-b737f7c41b06"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:55:15.520663 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:15.520639    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/635ba76a-39eb-4f14-8eb3-b737f7c41b06-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "635ba76a-39eb-4f14-8eb3-b737f7c41b06" (UID: "635ba76a-39eb-4f14-8eb3-b737f7c41b06"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:55:15.520981 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:15.520964    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/635ba76a-39eb-4f14-8eb3-b737f7c41b06-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "635ba76a-39eb-4f14-8eb3-b737f7c41b06" (UID: "635ba76a-39eb-4f14-8eb3-b737f7c41b06"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:55:15.522955 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:15.522937    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/635ba76a-39eb-4f14-8eb3-b737f7c41b06-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "635ba76a-39eb-4f14-8eb3-b737f7c41b06" (UID: "635ba76a-39eb-4f14-8eb3-b737f7c41b06"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:55:15.528684 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:15.528660    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/635ba76a-39eb-4f14-8eb3-b737f7c41b06-web-config" (OuterVolumeSpecName: "web-config") pod "635ba76a-39eb-4f14-8eb3-b737f7c41b06" (UID: "635ba76a-39eb-4f14-8eb3-b737f7c41b06"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:55:15.617379 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:15.617305    2579 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/635ba76a-39eb-4f14-8eb3-b737f7c41b06-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-138-224.ec2.internal\" DevicePath \"\""
Sep 09 20:55:15.617379 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:15.617338    2579 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/635ba76a-39eb-4f14-8eb3-b737f7c41b06-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-138-224.ec2.internal\" DevicePath \"\""
Sep 09 20:55:15.617379 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:15.617347    2579 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/635ba76a-39eb-4f14-8eb3-b737f7c41b06-config-out\") on node \"ip-10-0-138-224.ec2.internal\" DevicePath \"\""
Sep 09 20:55:15.617379 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:15.617370    2579 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/635ba76a-39eb-4f14-8eb3-b737f7c41b06-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-138-224.ec2.internal\" DevicePath \"\""
Sep 09 20:55:15.617379 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:15.617380    2579 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/635ba76a-39eb-4f14-8eb3-b737f7c41b06-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-138-224.ec2.internal\" DevicePath \"\""
Sep 09 20:55:15.617591 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:15.617389    2579 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/635ba76a-39eb-4f14-8eb3-b737f7c41b06-cluster-tls-config\") on node \"ip-10-0-138-224.ec2.internal\" DevicePath \"\""
Sep 09 20:55:15.617591 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:15.617413    2579 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/635ba76a-39eb-4f14-8eb3-b737f7c41b06-web-config\") on node \"ip-10-0-138-224.ec2.internal\" DevicePath \"\""
Sep 09 20:55:15.617591 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:15.617421    2579 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/635ba76a-39eb-4f14-8eb3-b737f7c41b06-config-volume\") on node \"ip-10-0-138-224.ec2.internal\" DevicePath \"\""
Sep 09 20:55:15.617591 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:15.617428    2579 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/635ba76a-39eb-4f14-8eb3-b737f7c41b06-tls-assets\") on node \"ip-10-0-138-224.ec2.internal\" DevicePath \"\""
Sep 09 20:55:15.617591 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:15.617435    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vpjkx\" (UniqueName: \"kubernetes.io/projected/635ba76a-39eb-4f14-8eb3-b737f7c41b06-kube-api-access-vpjkx\") on node \"ip-10-0-138-224.ec2.internal\" DevicePath \"\""
Sep 09 20:55:15.617591 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:15.617461    2579 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/635ba76a-39eb-4f14-8eb3-b737f7c41b06-secret-alertmanager-main-tls\") on node \"ip-10-0-138-224.ec2.internal\" DevicePath \"\""
Sep 09 20:55:15.617591 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:15.617468    2579 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/635ba76a-39eb-4f14-8eb3-b737f7c41b06-alertmanager-main-db\") on node \"ip-10-0-138-224.ec2.internal\" DevicePath \"\""
Sep 09 20:55:16.051947 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:16.051912    2579 generic.go:356] "Generic (PLEG): container finished" podID="635ba76a-39eb-4f14-8eb3-b737f7c41b06" containerID="b35ffad7008a0f1e0f496ee14fc6458285d9773a924d3ec291721325936338a2" exitCode=0
Sep 09 20:55:16.051947 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:16.051941    2579 generic.go:356] "Generic (PLEG): container finished" podID="635ba76a-39eb-4f14-8eb3-b737f7c41b06" containerID="87ad037a5d558c7a2bad84cc632b6882d5ffa038212f8ce068868d25fa38fa20" exitCode=0
Sep 09 20:55:16.052185 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:16.051958    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"635ba76a-39eb-4f14-8eb3-b737f7c41b06","Type":"ContainerDied","Data":"b35ffad7008a0f1e0f496ee14fc6458285d9773a924d3ec291721325936338a2"}
Sep 09 20:55:16.052185 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:16.051995    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:55:16.052185 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:16.052078    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"635ba76a-39eb-4f14-8eb3-b737f7c41b06","Type":"ContainerDied","Data":"87ad037a5d558c7a2bad84cc632b6882d5ffa038212f8ce068868d25fa38fa20"}
Sep 09 20:55:16.052185 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:16.052121    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"635ba76a-39eb-4f14-8eb3-b737f7c41b06","Type":"ContainerDied","Data":"4992849be9b78ceeeee2870ac35a84965c51726448de44a690257cbc14950da2"}
Sep 09 20:55:16.052185 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:16.052138    2579 scope.go:117] "RemoveContainer" containerID="60ad2131ec3eb6ea5cb3b6b301f3961c054cda597093e6eb87c318a74f47f4ca"
Sep 09 20:55:16.060408 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:16.060390    2579 scope.go:117] "RemoveContainer" containerID="b35ffad7008a0f1e0f496ee14fc6458285d9773a924d3ec291721325936338a2"
Sep 09 20:55:16.067064 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:16.067047    2579 scope.go:117] "RemoveContainer" containerID="ff902667b182e31061cbf572c763a246540db1f251944c72bc52f21fa2a4de7e"
Sep 09 20:55:16.073651 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:16.073633    2579 scope.go:117] "RemoveContainer" containerID="87ad037a5d558c7a2bad84cc632b6882d5ffa038212f8ce068868d25fa38fa20"
Sep 09 20:55:16.080009 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:16.079990    2579 scope.go:117] "RemoveContainer" containerID="dd8707e0b982d2b9518bc1e14edc4d96553cffd9f48ed5c403dc5d81f5e9e3eb"
Sep 09 20:55:16.081831 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:16.081811    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 09 20:55:16.087285 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:16.087266    2579 scope.go:117] "RemoveContainer" containerID="c7128cb2bbf82c3a0d9138b55dbc6c7d138ba13d48f0f7a77ead363c8ac4fdca"
Sep 09 20:55:16.087624 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:16.087605    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 09 20:55:16.094020 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:16.093999    2579 scope.go:117] "RemoveContainer" containerID="1b927781582bae493a70e2cfae0621b890de2ab8e093309a533ba46fc71098e7"
Sep 09 20:55:16.101023 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:16.101001    2579 scope.go:117] "RemoveContainer" containerID="60ad2131ec3eb6ea5cb3b6b301f3961c054cda597093e6eb87c318a74f47f4ca"
Sep 09 20:55:16.101266 ip-10-0-138-224 kubenswrapper[2579]: E0909 20:55:16.101249    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60ad2131ec3eb6ea5cb3b6b301f3961c054cda597093e6eb87c318a74f47f4ca\": container with ID starting with 60ad2131ec3eb6ea5cb3b6b301f3961c054cda597093e6eb87c318a74f47f4ca not found: ID does not exist" containerID="60ad2131ec3eb6ea5cb3b6b301f3961c054cda597093e6eb87c318a74f47f4ca"
Sep 09 20:55:16.101322 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:16.101275    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60ad2131ec3eb6ea5cb3b6b301f3961c054cda597093e6eb87c318a74f47f4ca"} err="failed to get container status \"60ad2131ec3eb6ea5cb3b6b301f3961c054cda597093e6eb87c318a74f47f4ca\": rpc error: code = NotFound desc = could not find container \"60ad2131ec3eb6ea5cb3b6b301f3961c054cda597093e6eb87c318a74f47f4ca\": container with ID starting with 60ad2131ec3eb6ea5cb3b6b301f3961c054cda597093e6eb87c318a74f47f4ca not found: ID does not exist"
Sep 09 20:55:16.101322 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:16.101291    2579 scope.go:117] "RemoveContainer" containerID="b35ffad7008a0f1e0f496ee14fc6458285d9773a924d3ec291721325936338a2"
Sep 09 20:55:16.101584 ip-10-0-138-224 kubenswrapper[2579]: E0909 20:55:16.101567    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b35ffad7008a0f1e0f496ee14fc6458285d9773a924d3ec291721325936338a2\": container with ID starting with b35ffad7008a0f1e0f496ee14fc6458285d9773a924d3ec291721325936338a2 not found: ID does not exist" containerID="b35ffad7008a0f1e0f496ee14fc6458285d9773a924d3ec291721325936338a2"
Sep 09 20:55:16.101636 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:16.101590    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b35ffad7008a0f1e0f496ee14fc6458285d9773a924d3ec291721325936338a2"} err="failed to get container status \"b35ffad7008a0f1e0f496ee14fc6458285d9773a924d3ec291721325936338a2\": rpc error: code = NotFound desc = could not find container \"b35ffad7008a0f1e0f496ee14fc6458285d9773a924d3ec291721325936338a2\": container with ID starting with b35ffad7008a0f1e0f496ee14fc6458285d9773a924d3ec291721325936338a2 not found: ID does not exist"
Sep 09 20:55:16.101636 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:16.101606    2579 scope.go:117] "RemoveContainer" containerID="ff902667b182e31061cbf572c763a246540db1f251944c72bc52f21fa2a4de7e"
Sep 09 20:55:16.101890 ip-10-0-138-224 kubenswrapper[2579]: E0909 20:55:16.101862    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff902667b182e31061cbf572c763a246540db1f251944c72bc52f21fa2a4de7e\": container with ID starting with ff902667b182e31061cbf572c763a246540db1f251944c72bc52f21fa2a4de7e not found: ID does not exist" containerID="ff902667b182e31061cbf572c763a246540db1f251944c72bc52f21fa2a4de7e"
Sep 09 20:55:16.101959 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:16.101894    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff902667b182e31061cbf572c763a246540db1f251944c72bc52f21fa2a4de7e"} err="failed to get container status \"ff902667b182e31061cbf572c763a246540db1f251944c72bc52f21fa2a4de7e\": rpc error: code = NotFound desc = could not find container \"ff902667b182e31061cbf572c763a246540db1f251944c72bc52f21fa2a4de7e\": container with ID starting with ff902667b182e31061cbf572c763a246540db1f251944c72bc52f21fa2a4de7e not found: ID does not exist"
Sep 09 20:55:16.101959 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:16.101907    2579 scope.go:117] "RemoveContainer" containerID="87ad037a5d558c7a2bad84cc632b6882d5ffa038212f8ce068868d25fa38fa20"
Sep 09 20:55:16.102160 ip-10-0-138-224 kubenswrapper[2579]: E0909 20:55:16.102145    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87ad037a5d558c7a2bad84cc632b6882d5ffa038212f8ce068868d25fa38fa20\": container with ID starting with 87ad037a5d558c7a2bad84cc632b6882d5ffa038212f8ce068868d25fa38fa20 not found: ID does not exist" containerID="87ad037a5d558c7a2bad84cc632b6882d5ffa038212f8ce068868d25fa38fa20"
Sep 09 20:55:16.102197 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:16.102162    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87ad037a5d558c7a2bad84cc632b6882d5ffa038212f8ce068868d25fa38fa20"} err="failed to get container status \"87ad037a5d558c7a2bad84cc632b6882d5ffa038212f8ce068868d25fa38fa20\": rpc error: code = NotFound desc = could not find container \"87ad037a5d558c7a2bad84cc632b6882d5ffa038212f8ce068868d25fa38fa20\": container with ID starting with 87ad037a5d558c7a2bad84cc632b6882d5ffa038212f8ce068868d25fa38fa20 not found: ID does not exist"
Sep 09 20:55:16.102197 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:16.102174    2579 scope.go:117] "RemoveContainer" containerID="dd8707e0b982d2b9518bc1e14edc4d96553cffd9f48ed5c403dc5d81f5e9e3eb"
Sep 09 20:55:16.102403 ip-10-0-138-224 kubenswrapper[2579]: E0909 20:55:16.102386    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd8707e0b982d2b9518bc1e14edc4d96553cffd9f48ed5c403dc5d81f5e9e3eb\": container with ID starting with dd8707e0b982d2b9518bc1e14edc4d96553cffd9f48ed5c403dc5d81f5e9e3eb not found: ID does not exist" containerID="dd8707e0b982d2b9518bc1e14edc4d96553cffd9f48ed5c403dc5d81f5e9e3eb"
Sep 09 20:55:16.102486 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:16.102408    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd8707e0b982d2b9518bc1e14edc4d96553cffd9f48ed5c403dc5d81f5e9e3eb"} err="failed to get container status \"dd8707e0b982d2b9518bc1e14edc4d96553cffd9f48ed5c403dc5d81f5e9e3eb\": rpc error: code = NotFound desc = could not find container \"dd8707e0b982d2b9518bc1e14edc4d96553cffd9f48ed5c403dc5d81f5e9e3eb\": container with ID starting with dd8707e0b982d2b9518bc1e14edc4d96553cffd9f48ed5c403dc5d81f5e9e3eb not found: ID does not exist"
Sep 09 20:55:16.102486 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:16.102426    2579 scope.go:117] "RemoveContainer" containerID="c7128cb2bbf82c3a0d9138b55dbc6c7d138ba13d48f0f7a77ead363c8ac4fdca"
Sep 09 20:55:16.102693 ip-10-0-138-224 kubenswrapper[2579]: E0909 20:55:16.102677    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7128cb2bbf82c3a0d9138b55dbc6c7d138ba13d48f0f7a77ead363c8ac4fdca\": container with ID starting with c7128cb2bbf82c3a0d9138b55dbc6c7d138ba13d48f0f7a77ead363c8ac4fdca not found: ID does not exist" containerID="c7128cb2bbf82c3a0d9138b55dbc6c7d138ba13d48f0f7a77ead363c8ac4fdca"
Sep 09 20:55:16.102737 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:16.102706    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7128cb2bbf82c3a0d9138b55dbc6c7d138ba13d48f0f7a77ead363c8ac4fdca"} err="failed to get container status \"c7128cb2bbf82c3a0d9138b55dbc6c7d138ba13d48f0f7a77ead363c8ac4fdca\": rpc error: code = NotFound desc = could not find container \"c7128cb2bbf82c3a0d9138b55dbc6c7d138ba13d48f0f7a77ead363c8ac4fdca\": container with ID starting with c7128cb2bbf82c3a0d9138b55dbc6c7d138ba13d48f0f7a77ead363c8ac4fdca not found: ID does not exist"
Sep 09 20:55:16.102737 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:16.102720    2579 scope.go:117] "RemoveContainer" containerID="1b927781582bae493a70e2cfae0621b890de2ab8e093309a533ba46fc71098e7"
Sep 09 20:55:16.102954 ip-10-0-138-224 kubenswrapper[2579]: E0909 20:55:16.102937    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b927781582bae493a70e2cfae0621b890de2ab8e093309a533ba46fc71098e7\": container with ID starting with 1b927781582bae493a70e2cfae0621b890de2ab8e093309a533ba46fc71098e7 not found: ID does not exist" containerID="1b927781582bae493a70e2cfae0621b890de2ab8e093309a533ba46fc71098e7"
Sep 09 20:55:16.103001 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:16.102961    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b927781582bae493a70e2cfae0621b890de2ab8e093309a533ba46fc71098e7"} err="failed to get container status \"1b927781582bae493a70e2cfae0621b890de2ab8e093309a533ba46fc71098e7\": rpc error: code = NotFound desc = could not find container \"1b927781582bae493a70e2cfae0621b890de2ab8e093309a533ba46fc71098e7\": container with ID starting with 1b927781582bae493a70e2cfae0621b890de2ab8e093309a533ba46fc71098e7 not found: ID does not exist"
Sep 09 20:55:16.103001 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:16.102974    2579 scope.go:117] "RemoveContainer" containerID="60ad2131ec3eb6ea5cb3b6b301f3961c054cda597093e6eb87c318a74f47f4ca"
Sep 09 20:55:16.103195 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:16.103177    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60ad2131ec3eb6ea5cb3b6b301f3961c054cda597093e6eb87c318a74f47f4ca"} err="failed to get container status \"60ad2131ec3eb6ea5cb3b6b301f3961c054cda597093e6eb87c318a74f47f4ca\": rpc error: code = NotFound desc = could not find container \"60ad2131ec3eb6ea5cb3b6b301f3961c054cda597093e6eb87c318a74f47f4ca\": container with ID starting with 60ad2131ec3eb6ea5cb3b6b301f3961c054cda597093e6eb87c318a74f47f4ca not found: ID does not exist"
Sep 09 20:55:16.103254 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:16.103197    2579 scope.go:117] "RemoveContainer" containerID="b35ffad7008a0f1e0f496ee14fc6458285d9773a924d3ec291721325936338a2"
Sep 09 20:55:16.103424 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:16.103404    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b35ffad7008a0f1e0f496ee14fc6458285d9773a924d3ec291721325936338a2"} err="failed to get container status \"b35ffad7008a0f1e0f496ee14fc6458285d9773a924d3ec291721325936338a2\": rpc error: code = NotFound desc = could not find container \"b35ffad7008a0f1e0f496ee14fc6458285d9773a924d3ec291721325936338a2\": container with ID starting with b35ffad7008a0f1e0f496ee14fc6458285d9773a924d3ec291721325936338a2 not found: ID does not exist"
Sep 09 20:55:16.103489 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:16.103425    2579 scope.go:117] "RemoveContainer" containerID="ff902667b182e31061cbf572c763a246540db1f251944c72bc52f21fa2a4de7e"
Sep 09 20:55:16.103694 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:16.103673    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff902667b182e31061cbf572c763a246540db1f251944c72bc52f21fa2a4de7e"} err="failed to get container status \"ff902667b182e31061cbf572c763a246540db1f251944c72bc52f21fa2a4de7e\": rpc error: code = NotFound desc = could not find container \"ff902667b182e31061cbf572c763a246540db1f251944c72bc52f21fa2a4de7e\": container with ID starting with ff902667b182e31061cbf572c763a246540db1f251944c72bc52f21fa2a4de7e not found: ID does not exist"
Sep 09 20:55:16.103772 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:16.103697    2579 scope.go:117] "RemoveContainer" containerID="87ad037a5d558c7a2bad84cc632b6882d5ffa038212f8ce068868d25fa38fa20"
Sep 09 20:55:16.103959 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:16.103940    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87ad037a5d558c7a2bad84cc632b6882d5ffa038212f8ce068868d25fa38fa20"} err="failed to get container status \"87ad037a5d558c7a2bad84cc632b6882d5ffa038212f8ce068868d25fa38fa20\": rpc error: code = NotFound desc = could not find container \"87ad037a5d558c7a2bad84cc632b6882d5ffa038212f8ce068868d25fa38fa20\": container with ID starting with 87ad037a5d558c7a2bad84cc632b6882d5ffa038212f8ce068868d25fa38fa20 not found: ID does not exist"
Sep 09 20:55:16.104011 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:16.103959    2579 scope.go:117] "RemoveContainer" containerID="dd8707e0b982d2b9518bc1e14edc4d96553cffd9f48ed5c403dc5d81f5e9e3eb"
Sep 09 20:55:16.104162 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:16.104144    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd8707e0b982d2b9518bc1e14edc4d96553cffd9f48ed5c403dc5d81f5e9e3eb"} err="failed to get container status \"dd8707e0b982d2b9518bc1e14edc4d96553cffd9f48ed5c403dc5d81f5e9e3eb\": rpc error: code = NotFound desc = could not find container \"dd8707e0b982d2b9518bc1e14edc4d96553cffd9f48ed5c403dc5d81f5e9e3eb\": container with ID starting with dd8707e0b982d2b9518bc1e14edc4d96553cffd9f48ed5c403dc5d81f5e9e3eb not found: ID does not exist"
Sep 09 20:55:16.104209 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:16.104162    2579 scope.go:117] "RemoveContainer" containerID="c7128cb2bbf82c3a0d9138b55dbc6c7d138ba13d48f0f7a77ead363c8ac4fdca"
Sep 09 20:55:16.104372 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:16.104353    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7128cb2bbf82c3a0d9138b55dbc6c7d138ba13d48f0f7a77ead363c8ac4fdca"} err="failed to get container status \"c7128cb2bbf82c3a0d9138b55dbc6c7d138ba13d48f0f7a77ead363c8ac4fdca\": rpc error: code = NotFound desc = could not find container \"c7128cb2bbf82c3a0d9138b55dbc6c7d138ba13d48f0f7a77ead363c8ac4fdca\": container with ID starting with c7128cb2bbf82c3a0d9138b55dbc6c7d138ba13d48f0f7a77ead363c8ac4fdca not found: ID does not exist"
Sep 09 20:55:16.104457 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:16.104373    2579 scope.go:117] "RemoveContainer" containerID="1b927781582bae493a70e2cfae0621b890de2ab8e093309a533ba46fc71098e7"
Sep 09 20:55:16.104639 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:16.104622    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b927781582bae493a70e2cfae0621b890de2ab8e093309a533ba46fc71098e7"} err="failed to get container status \"1b927781582bae493a70e2cfae0621b890de2ab8e093309a533ba46fc71098e7\": rpc error: code = NotFound desc = could not find container \"1b927781582bae493a70e2cfae0621b890de2ab8e093309a533ba46fc71098e7\": container with ID starting with 1b927781582bae493a70e2cfae0621b890de2ab8e093309a533ba46fc71098e7 not found: ID does not exist"
Sep 09 20:55:16.139285 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:16.139263    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 09 20:55:16.139545 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:16.139530    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="635ba76a-39eb-4f14-8eb3-b737f7c41b06" containerName="init-config-reloader"
Sep 09 20:55:16.139545 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:16.139545    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="635ba76a-39eb-4f14-8eb3-b737f7c41b06" containerName="init-config-reloader"
Sep 09 20:55:16.139665 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:16.139553    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="635ba76a-39eb-4f14-8eb3-b737f7c41b06" containerName="kube-rbac-proxy-metric"
Sep 09 20:55:16.139665 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:16.139558    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="635ba76a-39eb-4f14-8eb3-b737f7c41b06" containerName="kube-rbac-proxy-metric"
Sep 09 20:55:16.139665 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:16.139569    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f4343b3e-2a6c-41f8-95df-f9a50bcc3d8d" containerName="console"
Sep 09 20:55:16.139665 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:16.139574    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="f4343b3e-2a6c-41f8-95df-f9a50bcc3d8d" containerName="console"
Sep 09 20:55:16.139665 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:16.139580    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="635ba76a-39eb-4f14-8eb3-b737f7c41b06" containerName="prom-label-proxy"
Sep 09 20:55:16.139665 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:16.139604    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="635ba76a-39eb-4f14-8eb3-b737f7c41b06" containerName="prom-label-proxy"
Sep 09 20:55:16.139665 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:16.139613    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="635ba76a-39eb-4f14-8eb3-b737f7c41b06" containerName="alertmanager"
Sep 09 20:55:16.139665 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:16.139618    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="635ba76a-39eb-4f14-8eb3-b737f7c41b06" containerName="alertmanager"
Sep 09 20:55:16.139665 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:16.139630    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="635ba76a-39eb-4f14-8eb3-b737f7c41b06" containerName="kube-rbac-proxy"
Sep 09 20:55:16.139665 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:16.139634    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="635ba76a-39eb-4f14-8eb3-b737f7c41b06" containerName="kube-rbac-proxy"
Sep 09 20:55:16.139665 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:16.139641    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="635ba76a-39eb-4f14-8eb3-b737f7c41b06" containerName="config-reloader"
Sep 09 20:55:16.139665 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:16.139646    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="635ba76a-39eb-4f14-8eb3-b737f7c41b06" containerName="config-reloader"
Sep 09 20:55:16.139665 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:16.139653    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="635ba76a-39eb-4f14-8eb3-b737f7c41b06" containerName="kube-rbac-proxy-web"
Sep 09 20:55:16.139665 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:16.139657    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="635ba76a-39eb-4f14-8eb3-b737f7c41b06" containerName="kube-rbac-proxy-web"
Sep 09 20:55:16.140062 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:16.139701    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="f4343b3e-2a6c-41f8-95df-f9a50bcc3d8d" containerName="console"
Sep 09 20:55:16.140062 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:16.139707    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="635ba76a-39eb-4f14-8eb3-b737f7c41b06" containerName="kube-rbac-proxy"
Sep 09 20:55:16.140062 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:16.139713    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="635ba76a-39eb-4f14-8eb3-b737f7c41b06" containerName="kube-rbac-proxy-metric"
Sep 09 20:55:16.140062 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:16.139719    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="635ba76a-39eb-4f14-8eb3-b737f7c41b06" containerName="prom-label-proxy"
Sep 09 20:55:16.140062 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:16.139724    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="635ba76a-39eb-4f14-8eb3-b737f7c41b06" containerName="alertmanager"
Sep 09 20:55:16.140062 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:16.139729    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="635ba76a-39eb-4f14-8eb3-b737f7c41b06" containerName="config-reloader"
Sep 09 20:55:16.140062 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:16.139733    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="635ba76a-39eb-4f14-8eb3-b737f7c41b06" containerName="kube-rbac-proxy-web"
Sep 09 20:55:16.144613 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:16.144594    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:55:16.147430 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:16.147404    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 09 20:55:16.147532 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:16.147466    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 09 20:55:16.147591 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:16.147551    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 09 20:55:16.147690 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:16.147670    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 09 20:55:16.147932 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:16.147902    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 09 20:55:16.148022 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:16.147979    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 09 20:55:16.148089 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:16.147998    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-hh79h\""
Sep 09 20:55:16.148089 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:16.148066    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 09 20:55:16.148089 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:16.148081    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 09 20:55:16.167794 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:16.167752    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 09 20:55:16.168376 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:16.168360    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 09 20:55:16.323293 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:16.323198    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b3f6e9a2-a792-466e-8539-c769c5f74621-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"b3f6e9a2-a792-466e-8539-c769c5f74621\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:55:16.323293 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:16.323244    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t658w\" (UniqueName: \"kubernetes.io/projected/b3f6e9a2-a792-466e-8539-c769c5f74621-kube-api-access-t658w\") pod \"alertmanager-main-0\" (UID: \"b3f6e9a2-a792-466e-8539-c769c5f74621\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:55:16.323293 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:16.323264    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b3f6e9a2-a792-466e-8539-c769c5f74621-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"b3f6e9a2-a792-466e-8539-c769c5f74621\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:55:16.323293 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:16.323282    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b3f6e9a2-a792-466e-8539-c769c5f74621-tls-assets\") pod \"alertmanager-main-0\" (UID: \"b3f6e9a2-a792-466e-8539-c769c5f74621\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:55:16.323556 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:16.323331    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b3f6e9a2-a792-466e-8539-c769c5f74621-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"b3f6e9a2-a792-466e-8539-c769c5f74621\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:55:16.323556 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:16.323405    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b3f6e9a2-a792-466e-8539-c769c5f74621-web-config\") pod \"alertmanager-main-0\" (UID: \"b3f6e9a2-a792-466e-8539-c769c5f74621\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:55:16.323556 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:16.323424    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b3f6e9a2-a792-466e-8539-c769c5f74621-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"b3f6e9a2-a792-466e-8539-c769c5f74621\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:55:16.323556 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:16.323492    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b3f6e9a2-a792-466e-8539-c769c5f74621-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"b3f6e9a2-a792-466e-8539-c769c5f74621\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:55:16.323556 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:16.323513    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b3f6e9a2-a792-466e-8539-c769c5f74621-config-volume\") pod \"alertmanager-main-0\" (UID: \"b3f6e9a2-a792-466e-8539-c769c5f74621\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:55:16.323556 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:16.323533    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b3f6e9a2-a792-466e-8539-c769c5f74621-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"b3f6e9a2-a792-466e-8539-c769c5f74621\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:55:16.323726 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:16.323578    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b3f6e9a2-a792-466e-8539-c769c5f74621-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"b3f6e9a2-a792-466e-8539-c769c5f74621\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:55:16.323726 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:16.323623    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b3f6e9a2-a792-466e-8539-c769c5f74621-config-out\") pod \"alertmanager-main-0\" (UID: \"b3f6e9a2-a792-466e-8539-c769c5f74621\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:55:16.323726 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:16.323642    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b3f6e9a2-a792-466e-8539-c769c5f74621-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"b3f6e9a2-a792-466e-8539-c769c5f74621\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:55:16.424710 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:16.424680    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b3f6e9a2-a792-466e-8539-c769c5f74621-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"b3f6e9a2-a792-466e-8539-c769c5f74621\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:55:16.424710 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:16.424713    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b3f6e9a2-a792-466e-8539-c769c5f74621-config-volume\") pod \"alertmanager-main-0\" (UID: \"b3f6e9a2-a792-466e-8539-c769c5f74621\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:55:16.425112 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:16.424732    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b3f6e9a2-a792-466e-8539-c769c5f74621-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"b3f6e9a2-a792-466e-8539-c769c5f74621\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:55:16.425112 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:16.424841    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b3f6e9a2-a792-466e-8539-c769c5f74621-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"b3f6e9a2-a792-466e-8539-c769c5f74621\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:55:16.425112 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:16.424869    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b3f6e9a2-a792-466e-8539-c769c5f74621-config-out\") pod \"alertmanager-main-0\" (UID: \"b3f6e9a2-a792-466e-8539-c769c5f74621\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:55:16.425112 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:16.424899    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b3f6e9a2-a792-466e-8539-c769c5f74621-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"b3f6e9a2-a792-466e-8539-c769c5f74621\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:55:16.425112 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:16.424938    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b3f6e9a2-a792-466e-8539-c769c5f74621-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"b3f6e9a2-a792-466e-8539-c769c5f74621\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:55:16.425112 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:16.424967    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t658w\" (UniqueName: \"kubernetes.io/projected/b3f6e9a2-a792-466e-8539-c769c5f74621-kube-api-access-t658w\") pod \"alertmanager-main-0\" (UID: \"b3f6e9a2-a792-466e-8539-c769c5f74621\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:55:16.425112 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:16.424997    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b3f6e9a2-a792-466e-8539-c769c5f74621-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"b3f6e9a2-a792-466e-8539-c769c5f74621\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:55:16.425112 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:16.425022    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b3f6e9a2-a792-466e-8539-c769c5f74621-tls-assets\") pod \"alertmanager-main-0\" (UID: \"b3f6e9a2-a792-466e-8539-c769c5f74621\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:55:16.425112 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:16.425052    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b3f6e9a2-a792-466e-8539-c769c5f74621-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"b3f6e9a2-a792-466e-8539-c769c5f74621\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:55:16.425112 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:16.425079    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b3f6e9a2-a792-466e-8539-c769c5f74621-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"b3f6e9a2-a792-466e-8539-c769c5f74621\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:55:16.425112 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:16.425083    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b3f6e9a2-a792-466e-8539-c769c5f74621-web-config\") pod \"alertmanager-main-0\" (UID: \"b3f6e9a2-a792-466e-8539-c769c5f74621\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:55:16.425836 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:16.425144    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b3f6e9a2-a792-466e-8539-c769c5f74621-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"b3f6e9a2-a792-466e-8539-c769c5f74621\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:55:16.426115 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:16.426088    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b3f6e9a2-a792-466e-8539-c769c5f74621-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"b3f6e9a2-a792-466e-8539-c769c5f74621\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:55:16.427919 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:16.427885    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b3f6e9a2-a792-466e-8539-c769c5f74621-config-out\") pod \"alertmanager-main-0\" (UID: \"b3f6e9a2-a792-466e-8539-c769c5f74621\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:55:16.428255 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:16.428226    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b3f6e9a2-a792-466e-8539-c769c5f74621-config-volume\") pod \"alertmanager-main-0\" (UID: \"b3f6e9a2-a792-466e-8539-c769c5f74621\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:55:16.428350 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:16.428286    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b3f6e9a2-a792-466e-8539-c769c5f74621-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"b3f6e9a2-a792-466e-8539-c769c5f74621\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:55:16.428957 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:16.428668    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b3f6e9a2-a792-466e-8539-c769c5f74621-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"b3f6e9a2-a792-466e-8539-c769c5f74621\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:55:16.428957 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:16.428729    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b3f6e9a2-a792-466e-8539-c769c5f74621-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"b3f6e9a2-a792-466e-8539-c769c5f74621\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:55:16.428957 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:16.428743    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b3f6e9a2-a792-466e-8539-c769c5f74621-web-config\") pod \"alertmanager-main-0\" (UID: \"b3f6e9a2-a792-466e-8539-c769c5f74621\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:55:16.428957 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:16.428845    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b3f6e9a2-a792-466e-8539-c769c5f74621-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"b3f6e9a2-a792-466e-8539-c769c5f74621\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:55:16.428957 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:16.428903    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b3f6e9a2-a792-466e-8539-c769c5f74621-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"b3f6e9a2-a792-466e-8539-c769c5f74621\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:55:16.429225 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:16.429092    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b3f6e9a2-a792-466e-8539-c769c5f74621-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"b3f6e9a2-a792-466e-8539-c769c5f74621\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:55:16.430374 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:16.430357    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b3f6e9a2-a792-466e-8539-c769c5f74621-tls-assets\") pod \"alertmanager-main-0\" (UID: \"b3f6e9a2-a792-466e-8539-c769c5f74621\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:55:16.439262 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:16.439238    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t658w\" (UniqueName: \"kubernetes.io/projected/b3f6e9a2-a792-466e-8539-c769c5f74621-kube-api-access-t658w\") pod \"alertmanager-main-0\" (UID: \"b3f6e9a2-a792-466e-8539-c769c5f74621\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:55:16.454154 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:16.454138    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:55:16.737625 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:16.737597    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 09 20:55:16.738366 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:55:16.738342    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3f6e9a2_a792_466e_8539_c769c5f74621.slice/crio-cefff88fba776413dad0e6e8683995a450e22504600b9b147f9bc8bc762be80c WatchSource:0}: Error finding container cefff88fba776413dad0e6e8683995a450e22504600b9b147f9bc8bc762be80c: Status 404 returned error can't find the container with id cefff88fba776413dad0e6e8683995a450e22504600b9b147f9bc8bc762be80c
Sep 09 20:55:17.057616 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:17.057522    2579 generic.go:356] "Generic (PLEG): container finished" podID="b3f6e9a2-a792-466e-8539-c769c5f74621" containerID="2fca98ebd44728abdf4da30a3e68028a2b5b17a53e90dde0ec60cbc3378091e1" exitCode=0
Sep 09 20:55:17.057616 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:17.057569    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b3f6e9a2-a792-466e-8539-c769c5f74621","Type":"ContainerDied","Data":"2fca98ebd44728abdf4da30a3e68028a2b5b17a53e90dde0ec60cbc3378091e1"}
Sep 09 20:55:17.057799 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:17.057638    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b3f6e9a2-a792-466e-8539-c769c5f74621","Type":"ContainerStarted","Data":"cefff88fba776413dad0e6e8683995a450e22504600b9b147f9bc8bc762be80c"}
Sep 09 20:55:17.290331 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:17.290308    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="635ba76a-39eb-4f14-8eb3-b737f7c41b06" path="/var/lib/kubelet/pods/635ba76a-39eb-4f14-8eb3-b737f7c41b06/volumes"
Sep 09 20:55:18.064495 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:18.064434    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b3f6e9a2-a792-466e-8539-c769c5f74621","Type":"ContainerStarted","Data":"198cbb49b38ca71406d5cb2dca3746a342f4c0bc4c183c4d69a27c57fcf3b754"}
Sep 09 20:55:18.064869 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:18.064526    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b3f6e9a2-a792-466e-8539-c769c5f74621","Type":"ContainerStarted","Data":"8cda0b4ee4811436b7b1f3d79680efe718b03ea23a14f1f0290691c5e9dafb82"}
Sep 09 20:55:18.064869 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:18.064560    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b3f6e9a2-a792-466e-8539-c769c5f74621","Type":"ContainerStarted","Data":"00c46231084ea87916f2a92dae59f2bc6520a3ad4a96b5c87a84f790d6ad4dfb"}
Sep 09 20:55:18.064869 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:18.064584    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b3f6e9a2-a792-466e-8539-c769c5f74621","Type":"ContainerStarted","Data":"495661e8a7db4d3bf80181c0e8abb6de3fe615799dd93b059aef7104788fc3f0"}
Sep 09 20:55:18.064869 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:18.064606    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b3f6e9a2-a792-466e-8539-c769c5f74621","Type":"ContainerStarted","Data":"50d98145d00984ece464ee48d036dc07032a2f8948974ea65eaf58faedfacabb"}
Sep 09 20:55:18.064869 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:18.064629    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b3f6e9a2-a792-466e-8539-c769c5f74621","Type":"ContainerStarted","Data":"9d367a9fffa2ca4e1eb61e4230df7293234d3b7076ad3f635abbe1adad4a5487"}
Sep 09 20:55:18.096232 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:18.096172    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.08065719 podStartE2EDuration="2.096151356s" totalImagesPullingTime="0s" totalInitContainerRuntime="15.494166ms" isStatefulPod=false podCreationTimestamp="2026-09-09 20:55:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:55:18.093937138 +0000 UTC m=+175.334031281" watchObservedRunningTime="2026-09-09 20:55:18.096151356 +0000 UTC m=+175.336245475"
Sep 09 20:55:18.196562 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:18.196531    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-8664f699df-gpz7g"]
Sep 09 20:55:18.199996 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:18.199976    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-8664f699df-gpz7g"
Sep 09 20:55:18.205589 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:18.205563    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Sep 09 20:55:18.205821 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:18.205802    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Sep 09 20:55:18.206740 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:18.206629    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Sep 09 20:55:18.206970 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:18.206949    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Sep 09 20:55:18.207034 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:18.206969    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Sep 09 20:55:18.207340 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:18.207326    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-srsgn\""
Sep 09 20:55:18.212163 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:18.212131    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Sep 09 20:55:18.225106 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:18.225079    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-8664f699df-gpz7g"]
Sep 09 20:55:18.342765 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:18.342675    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/7c4733c8-b3da-4ef5-92a6-b0c00921e605-telemeter-client-tls\") pod \"telemeter-client-8664f699df-gpz7g\" (UID: \"7c4733c8-b3da-4ef5-92a6-b0c00921e605\") " pod="openshift-monitoring/telemeter-client-8664f699df-gpz7g"
Sep 09 20:55:18.342765 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:18.342729    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7c4733c8-b3da-4ef5-92a6-b0c00921e605-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-8664f699df-gpz7g\" (UID: \"7c4733c8-b3da-4ef5-92a6-b0c00921e605\") " pod="openshift-monitoring/telemeter-client-8664f699df-gpz7g"
Sep 09 20:55:18.342765 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:18.342752    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/7c4733c8-b3da-4ef5-92a6-b0c00921e605-federate-client-tls\") pod \"telemeter-client-8664f699df-gpz7g\" (UID: \"7c4733c8-b3da-4ef5-92a6-b0c00921e605\") " pod="openshift-monitoring/telemeter-client-8664f699df-gpz7g"
Sep 09 20:55:18.343037 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:18.342793    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dnhgs\" (UniqueName: \"kubernetes.io/projected/7c4733c8-b3da-4ef5-92a6-b0c00921e605-kube-api-access-dnhgs\") pod \"telemeter-client-8664f699df-gpz7g\" (UID: \"7c4733c8-b3da-4ef5-92a6-b0c00921e605\") " pod="openshift-monitoring/telemeter-client-8664f699df-gpz7g"
Sep 09 20:55:18.343037 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:18.342816    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7c4733c8-b3da-4ef5-92a6-b0c00921e605-metrics-client-ca\") pod \"telemeter-client-8664f699df-gpz7g\" (UID: \"7c4733c8-b3da-4ef5-92a6-b0c00921e605\") " pod="openshift-monitoring/telemeter-client-8664f699df-gpz7g"
Sep 09 20:55:18.343037 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:18.342849    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c4733c8-b3da-4ef5-92a6-b0c00921e605-serving-certs-ca-bundle\") pod \"telemeter-client-8664f699df-gpz7g\" (UID: \"7c4733c8-b3da-4ef5-92a6-b0c00921e605\") " pod="openshift-monitoring/telemeter-client-8664f699df-gpz7g"
Sep 09 20:55:18.343037 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:18.342892    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c4733c8-b3da-4ef5-92a6-b0c00921e605-telemeter-trusted-ca-bundle\") pod \"telemeter-client-8664f699df-gpz7g\" (UID: \"7c4733c8-b3da-4ef5-92a6-b0c00921e605\") " pod="openshift-monitoring/telemeter-client-8664f699df-gpz7g"
Sep 09 20:55:18.343037 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:18.342945    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/7c4733c8-b3da-4ef5-92a6-b0c00921e605-secret-telemeter-client\") pod \"telemeter-client-8664f699df-gpz7g\" (UID: \"7c4733c8-b3da-4ef5-92a6-b0c00921e605\") " pod="openshift-monitoring/telemeter-client-8664f699df-gpz7g"
Sep 09 20:55:18.443985 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:18.443947    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7c4733c8-b3da-4ef5-92a6-b0c00921e605-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-8664f699df-gpz7g\" (UID: \"7c4733c8-b3da-4ef5-92a6-b0c00921e605\") " pod="openshift-monitoring/telemeter-client-8664f699df-gpz7g"
Sep 09 20:55:18.443985 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:18.443987    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/7c4733c8-b3da-4ef5-92a6-b0c00921e605-federate-client-tls\") pod \"telemeter-client-8664f699df-gpz7g\" (UID: \"7c4733c8-b3da-4ef5-92a6-b0c00921e605\") " pod="openshift-monitoring/telemeter-client-8664f699df-gpz7g"
Sep 09 20:55:18.444185 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:18.444012    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnhgs\" (UniqueName: \"kubernetes.io/projected/7c4733c8-b3da-4ef5-92a6-b0c00921e605-kube-api-access-dnhgs\") pod \"telemeter-client-8664f699df-gpz7g\" (UID: \"7c4733c8-b3da-4ef5-92a6-b0c00921e605\") " pod="openshift-monitoring/telemeter-client-8664f699df-gpz7g"
Sep 09 20:55:18.444185 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:18.444136    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7c4733c8-b3da-4ef5-92a6-b0c00921e605-metrics-client-ca\") pod \"telemeter-client-8664f699df-gpz7g\" (UID: \"7c4733c8-b3da-4ef5-92a6-b0c00921e605\") " pod="openshift-monitoring/telemeter-client-8664f699df-gpz7g"
Sep 09 20:55:18.444295 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:18.444209    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c4733c8-b3da-4ef5-92a6-b0c00921e605-serving-certs-ca-bundle\") pod \"telemeter-client-8664f699df-gpz7g\" (UID: \"7c4733c8-b3da-4ef5-92a6-b0c00921e605\") " pod="openshift-monitoring/telemeter-client-8664f699df-gpz7g"
Sep 09 20:55:18.444295 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:18.444243    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c4733c8-b3da-4ef5-92a6-b0c00921e605-telemeter-trusted-ca-bundle\") pod \"telemeter-client-8664f699df-gpz7g\" (UID: \"7c4733c8-b3da-4ef5-92a6-b0c00921e605\") " pod="openshift-monitoring/telemeter-client-8664f699df-gpz7g"
Sep 09 20:55:18.444295 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:18.444268    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/7c4733c8-b3da-4ef5-92a6-b0c00921e605-secret-telemeter-client\") pod \"telemeter-client-8664f699df-gpz7g\" (UID: \"7c4733c8-b3da-4ef5-92a6-b0c00921e605\") " pod="openshift-monitoring/telemeter-client-8664f699df-gpz7g"
Sep 09 20:55:18.444554 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:18.444317    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/7c4733c8-b3da-4ef5-92a6-b0c00921e605-telemeter-client-tls\") pod \"telemeter-client-8664f699df-gpz7g\" (UID: \"7c4733c8-b3da-4ef5-92a6-b0c00921e605\") " pod="openshift-monitoring/telemeter-client-8664f699df-gpz7g"
Sep 09 20:55:18.444933 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:18.444911    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7c4733c8-b3da-4ef5-92a6-b0c00921e605-metrics-client-ca\") pod \"telemeter-client-8664f699df-gpz7g\" (UID: \"7c4733c8-b3da-4ef5-92a6-b0c00921e605\") " pod="openshift-monitoring/telemeter-client-8664f699df-gpz7g"
Sep 09 20:55:18.445058 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:18.445025    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c4733c8-b3da-4ef5-92a6-b0c00921e605-serving-certs-ca-bundle\") pod \"telemeter-client-8664f699df-gpz7g\" (UID: \"7c4733c8-b3da-4ef5-92a6-b0c00921e605\") " pod="openshift-monitoring/telemeter-client-8664f699df-gpz7g"
Sep 09 20:55:18.445347 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:18.445221    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c4733c8-b3da-4ef5-92a6-b0c00921e605-telemeter-trusted-ca-bundle\") pod \"telemeter-client-8664f699df-gpz7g\" (UID: \"7c4733c8-b3da-4ef5-92a6-b0c00921e605\") " pod="openshift-monitoring/telemeter-client-8664f699df-gpz7g"
Sep 09 20:55:18.446688 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:18.446662    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/7c4733c8-b3da-4ef5-92a6-b0c00921e605-federate-client-tls\") pod \"telemeter-client-8664f699df-gpz7g\" (UID: \"7c4733c8-b3da-4ef5-92a6-b0c00921e605\") " pod="openshift-monitoring/telemeter-client-8664f699df-gpz7g"
Sep 09 20:55:18.446995 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:18.446966    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/7c4733c8-b3da-4ef5-92a6-b0c00921e605-secret-telemeter-client\") pod \"telemeter-client-8664f699df-gpz7g\" (UID: \"7c4733c8-b3da-4ef5-92a6-b0c00921e605\") " pod="openshift-monitoring/telemeter-client-8664f699df-gpz7g"
Sep 09 20:55:18.447073 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:18.447054    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7c4733c8-b3da-4ef5-92a6-b0c00921e605-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-8664f699df-gpz7g\" (UID: \"7c4733c8-b3da-4ef5-92a6-b0c00921e605\") " pod="openshift-monitoring/telemeter-client-8664f699df-gpz7g"
Sep 09 20:55:18.447219 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:18.447202    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/7c4733c8-b3da-4ef5-92a6-b0c00921e605-telemeter-client-tls\") pod \"telemeter-client-8664f699df-gpz7g\" (UID: \"7c4733c8-b3da-4ef5-92a6-b0c00921e605\") " pod="openshift-monitoring/telemeter-client-8664f699df-gpz7g"
Sep 09 20:55:18.453371 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:18.453350    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dnhgs\" (UniqueName: \"kubernetes.io/projected/7c4733c8-b3da-4ef5-92a6-b0c00921e605-kube-api-access-dnhgs\") pod \"telemeter-client-8664f699df-gpz7g\" (UID: \"7c4733c8-b3da-4ef5-92a6-b0c00921e605\") " pod="openshift-monitoring/telemeter-client-8664f699df-gpz7g"
Sep 09 20:55:18.509209 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:18.509174    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-8664f699df-gpz7g"
Sep 09 20:55:18.814700 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:18.814657    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-8664f699df-gpz7g"]
Sep 09 20:55:18.818653 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:55:18.818606    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c4733c8_b3da_4ef5_92a6_b0c00921e605.slice/crio-133e910e229cd616dbb88b52008d805c36d57163c27975239b3239892691da81 WatchSource:0}: Error finding container 133e910e229cd616dbb88b52008d805c36d57163c27975239b3239892691da81: Status 404 returned error can't find the container with id 133e910e229cd616dbb88b52008d805c36d57163c27975239b3239892691da81
Sep 09 20:55:19.069956 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:19.069874    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-8664f699df-gpz7g" event={"ID":"7c4733c8-b3da-4ef5-92a6-b0c00921e605","Type":"ContainerStarted","Data":"133e910e229cd616dbb88b52008d805c36d57163c27975239b3239892691da81"}
Sep 09 20:55:21.079862 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:21.079814    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-8664f699df-gpz7g" event={"ID":"7c4733c8-b3da-4ef5-92a6-b0c00921e605","Type":"ContainerStarted","Data":"ca44db26ff788e2052bd8a3a130be536945acb7381351973d23418a6c5c7cdd5"}
Sep 09 20:55:21.080304 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:21.079882    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-8664f699df-gpz7g" event={"ID":"7c4733c8-b3da-4ef5-92a6-b0c00921e605","Type":"ContainerStarted","Data":"6fd8a6836285d572350b0e6c28c74584ccabf930661d1c4ea95e965f01bffbd7"}
Sep 09 20:55:21.080304 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:21.079901    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-8664f699df-gpz7g" event={"ID":"7c4733c8-b3da-4ef5-92a6-b0c00921e605","Type":"ContainerStarted","Data":"3366075b52d5758c03704ad8675b012f1969d66c90119a75942163d572b8668e"}
Sep 09 20:55:21.104497 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:21.104417    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-8664f699df-gpz7g" podStartSLOduration=1.046472663 podStartE2EDuration="3.104400076s" totalImagesPullingTime="2.057927413s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:55:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:55:21.103634456 +0000 UTC m=+178.343728596" watchObservedRunningTime="2026-09-09 20:55:21.104400076 +0000 UTC m=+178.344494224"
Sep 09 20:55:21.174916 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:21.174881    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-549b6ccc54-7n8jr"]
Sep 09 20:55:21.178230 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:21.178208    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-549b6ccc54-7n8jr"
Sep 09 20:55:21.190935 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:21.190910    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-549b6ccc54-7n8jr"]
Sep 09 20:55:21.269279 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:21.269233    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/eb40731c-836a-454f-9dad-af5808c13ba8-oauth-serving-cert\") pod \"console-549b6ccc54-7n8jr\" (UID: \"eb40731c-836a-454f-9dad-af5808c13ba8\") " pod="openshift-console/console-549b6ccc54-7n8jr"
Sep 09 20:55:21.269279 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:21.269280    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/eb40731c-836a-454f-9dad-af5808c13ba8-console-config\") pod \"console-549b6ccc54-7n8jr\" (UID: \"eb40731c-836a-454f-9dad-af5808c13ba8\") " pod="openshift-console/console-549b6ccc54-7n8jr"
Sep 09 20:55:21.269547 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:21.269302    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/eb40731c-836a-454f-9dad-af5808c13ba8-console-serving-cert\") pod \"console-549b6ccc54-7n8jr\" (UID: \"eb40731c-836a-454f-9dad-af5808c13ba8\") " pod="openshift-console/console-549b6ccc54-7n8jr"
Sep 09 20:55:21.269547 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:21.269344    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/eb40731c-836a-454f-9dad-af5808c13ba8-service-ca\") pod \"console-549b6ccc54-7n8jr\" (UID: \"eb40731c-836a-454f-9dad-af5808c13ba8\") " pod="openshift-console/console-549b6ccc54-7n8jr"
Sep 09 20:55:21.269547 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:21.269381    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eb40731c-836a-454f-9dad-af5808c13ba8-trusted-ca-bundle\") pod \"console-549b6ccc54-7n8jr\" (UID: \"eb40731c-836a-454f-9dad-af5808c13ba8\") " pod="openshift-console/console-549b6ccc54-7n8jr"
Sep 09 20:55:21.269547 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:21.269405    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sh642\" (UniqueName: \"kubernetes.io/projected/eb40731c-836a-454f-9dad-af5808c13ba8-kube-api-access-sh642\") pod \"console-549b6ccc54-7n8jr\" (UID: \"eb40731c-836a-454f-9dad-af5808c13ba8\") " pod="openshift-console/console-549b6ccc54-7n8jr"
Sep 09 20:55:21.269547 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:21.269487    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/eb40731c-836a-454f-9dad-af5808c13ba8-console-oauth-config\") pod \"console-549b6ccc54-7n8jr\" (UID: \"eb40731c-836a-454f-9dad-af5808c13ba8\") " pod="openshift-console/console-549b6ccc54-7n8jr"
Sep 09 20:55:21.370471 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:21.370404    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/eb40731c-836a-454f-9dad-af5808c13ba8-oauth-serving-cert\") pod \"console-549b6ccc54-7n8jr\" (UID: \"eb40731c-836a-454f-9dad-af5808c13ba8\") " pod="openshift-console/console-549b6ccc54-7n8jr"
Sep 09 20:55:21.370471 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:21.370477    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/eb40731c-836a-454f-9dad-af5808c13ba8-console-config\") pod \"console-549b6ccc54-7n8jr\" (UID: \"eb40731c-836a-454f-9dad-af5808c13ba8\") " pod="openshift-console/console-549b6ccc54-7n8jr"
Sep 09 20:55:21.370702 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:21.370507    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/eb40731c-836a-454f-9dad-af5808c13ba8-console-serving-cert\") pod \"console-549b6ccc54-7n8jr\" (UID: \"eb40731c-836a-454f-9dad-af5808c13ba8\") " pod="openshift-console/console-549b6ccc54-7n8jr"
Sep 09 20:55:21.370702 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:21.370524    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/eb40731c-836a-454f-9dad-af5808c13ba8-service-ca\") pod \"console-549b6ccc54-7n8jr\" (UID: \"eb40731c-836a-454f-9dad-af5808c13ba8\") " pod="openshift-console/console-549b6ccc54-7n8jr"
Sep 09 20:55:21.370702 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:21.370543    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eb40731c-836a-454f-9dad-af5808c13ba8-trusted-ca-bundle\") pod \"console-549b6ccc54-7n8jr\" (UID: \"eb40731c-836a-454f-9dad-af5808c13ba8\") " pod="openshift-console/console-549b6ccc54-7n8jr"
Sep 09 20:55:21.370702 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:21.370565    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sh642\" (UniqueName: \"kubernetes.io/projected/eb40731c-836a-454f-9dad-af5808c13ba8-kube-api-access-sh642\") pod \"console-549b6ccc54-7n8jr\" (UID: \"eb40731c-836a-454f-9dad-af5808c13ba8\") " pod="openshift-console/console-549b6ccc54-7n8jr"
Sep 09 20:55:21.370702 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:21.370629    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/eb40731c-836a-454f-9dad-af5808c13ba8-console-oauth-config\") pod \"console-549b6ccc54-7n8jr\" (UID: \"eb40731c-836a-454f-9dad-af5808c13ba8\") " pod="openshift-console/console-549b6ccc54-7n8jr"
Sep 09 20:55:21.371268 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:21.371238    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/eb40731c-836a-454f-9dad-af5808c13ba8-service-ca\") pod \"console-549b6ccc54-7n8jr\" (UID: \"eb40731c-836a-454f-9dad-af5808c13ba8\") " pod="openshift-console/console-549b6ccc54-7n8jr"
Sep 09 20:55:21.371380 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:21.371243    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/eb40731c-836a-454f-9dad-af5808c13ba8-oauth-serving-cert\") pod \"console-549b6ccc54-7n8jr\" (UID: \"eb40731c-836a-454f-9dad-af5808c13ba8\") " pod="openshift-console/console-549b6ccc54-7n8jr"
Sep 09 20:55:21.371380 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:21.371361    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/eb40731c-836a-454f-9dad-af5808c13ba8-console-config\") pod \"console-549b6ccc54-7n8jr\" (UID: \"eb40731c-836a-454f-9dad-af5808c13ba8\") " pod="openshift-console/console-549b6ccc54-7n8jr"
Sep 09 20:55:21.371542 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:21.371524    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eb40731c-836a-454f-9dad-af5808c13ba8-trusted-ca-bundle\") pod \"console-549b6ccc54-7n8jr\" (UID: \"eb40731c-836a-454f-9dad-af5808c13ba8\") " pod="openshift-console/console-549b6ccc54-7n8jr"
Sep 09 20:55:21.373189 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:21.373166    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/eb40731c-836a-454f-9dad-af5808c13ba8-console-oauth-config\") pod \"console-549b6ccc54-7n8jr\" (UID: \"eb40731c-836a-454f-9dad-af5808c13ba8\") " pod="openshift-console/console-549b6ccc54-7n8jr"
Sep 09 20:55:21.373268 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:21.373228    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/eb40731c-836a-454f-9dad-af5808c13ba8-console-serving-cert\") pod \"console-549b6ccc54-7n8jr\" (UID: \"eb40731c-836a-454f-9dad-af5808c13ba8\") " pod="openshift-console/console-549b6ccc54-7n8jr"
Sep 09 20:55:21.378827 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:21.378798    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sh642\" (UniqueName: \"kubernetes.io/projected/eb40731c-836a-454f-9dad-af5808c13ba8-kube-api-access-sh642\") pod \"console-549b6ccc54-7n8jr\" (UID: \"eb40731c-836a-454f-9dad-af5808c13ba8\") " pod="openshift-console/console-549b6ccc54-7n8jr"
Sep 09 20:55:21.486594 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:21.486508    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-549b6ccc54-7n8jr"
Sep 09 20:55:21.759282 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:21.759192    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-549b6ccc54-7n8jr"]
Sep 09 20:55:21.761640 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:55:21.761615    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb40731c_836a_454f_9dad_af5808c13ba8.slice/crio-a994be64959fb3984dc9fca66cbd332829da101917f942fd771f1acd9bbbc263 WatchSource:0}: Error finding container a994be64959fb3984dc9fca66cbd332829da101917f942fd771f1acd9bbbc263: Status 404 returned error can't find the container with id a994be64959fb3984dc9fca66cbd332829da101917f942fd771f1acd9bbbc263
Sep 09 20:55:22.086126 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:22.086031    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-549b6ccc54-7n8jr" event={"ID":"eb40731c-836a-454f-9dad-af5808c13ba8","Type":"ContainerStarted","Data":"69d1043fe71368c7dfa1baf806fa2ae753b250902d34f79dad8984d33661a183"}
Sep 09 20:55:22.086126 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:22.086099    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-549b6ccc54-7n8jr" event={"ID":"eb40731c-836a-454f-9dad-af5808c13ba8","Type":"ContainerStarted","Data":"a994be64959fb3984dc9fca66cbd332829da101917f942fd771f1acd9bbbc263"}
Sep 09 20:55:22.103480 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:22.103415    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-549b6ccc54-7n8jr" podStartSLOduration=1.103401727 podStartE2EDuration="1.103401727s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:55:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:55:22.102490467 +0000 UTC m=+179.342584587" watchObservedRunningTime="2026-09-09 20:55:22.103401727 +0000 UTC m=+179.343495845"
Sep 09 20:55:31.487115 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:31.487067    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-549b6ccc54-7n8jr"
Sep 09 20:55:31.487600 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:31.487151    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-549b6ccc54-7n8jr"
Sep 09 20:55:31.492013 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:31.491989    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-549b6ccc54-7n8jr"
Sep 09 20:55:32.130794 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:32.130765    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-549b6ccc54-7n8jr"
Sep 09 20:55:32.203109 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:32.203075    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5495fd7677-lg774"]
Sep 09 20:55:57.223925 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:57.223880    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5495fd7677-lg774" podUID="83dac517-6dc0-4404-b8e5-77b11d91cd9a" containerName="console" containerID="cri-o://2b9291d518c957dc3e5384cd8720fd8c6f394587b52e5bf826ecdd9bc6a978d9" gracePeriod=15
Sep 09 20:55:57.464048 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:57.464021    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5495fd7677-lg774"
Sep 09 20:55:57.554607 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:57.554544    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/83dac517-6dc0-4404-b8e5-77b11d91cd9a-console-serving-cert\") pod \"83dac517-6dc0-4404-b8e5-77b11d91cd9a\" (UID: \"83dac517-6dc0-4404-b8e5-77b11d91cd9a\") "
Sep 09 20:55:57.554607 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:57.554584    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/83dac517-6dc0-4404-b8e5-77b11d91cd9a-trusted-ca-bundle\") pod \"83dac517-6dc0-4404-b8e5-77b11d91cd9a\" (UID: \"83dac517-6dc0-4404-b8e5-77b11d91cd9a\") "
Sep 09 20:55:57.554768 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:57.554625    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/83dac517-6dc0-4404-b8e5-77b11d91cd9a-console-config\") pod \"83dac517-6dc0-4404-b8e5-77b11d91cd9a\" (UID: \"83dac517-6dc0-4404-b8e5-77b11d91cd9a\") "
Sep 09 20:55:57.554768 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:57.554666    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/83dac517-6dc0-4404-b8e5-77b11d91cd9a-service-ca\") pod \"83dac517-6dc0-4404-b8e5-77b11d91cd9a\" (UID: \"83dac517-6dc0-4404-b8e5-77b11d91cd9a\") "
Sep 09 20:55:57.554768 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:57.554702    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/83dac517-6dc0-4404-b8e5-77b11d91cd9a-oauth-serving-cert\") pod \"83dac517-6dc0-4404-b8e5-77b11d91cd9a\" (UID: \"83dac517-6dc0-4404-b8e5-77b11d91cd9a\") "
Sep 09 20:55:57.554768 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:57.554732    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pcddq\" (UniqueName: \"kubernetes.io/projected/83dac517-6dc0-4404-b8e5-77b11d91cd9a-kube-api-access-pcddq\") pod \"83dac517-6dc0-4404-b8e5-77b11d91cd9a\" (UID: \"83dac517-6dc0-4404-b8e5-77b11d91cd9a\") "
Sep 09 20:55:57.554768 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:57.554764    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/83dac517-6dc0-4404-b8e5-77b11d91cd9a-console-oauth-config\") pod \"83dac517-6dc0-4404-b8e5-77b11d91cd9a\" (UID: \"83dac517-6dc0-4404-b8e5-77b11d91cd9a\") "
Sep 09 20:55:57.555124 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:57.555084    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83dac517-6dc0-4404-b8e5-77b11d91cd9a-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "83dac517-6dc0-4404-b8e5-77b11d91cd9a" (UID: "83dac517-6dc0-4404-b8e5-77b11d91cd9a"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:55:57.555124 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:57.555095    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83dac517-6dc0-4404-b8e5-77b11d91cd9a-console-config" (OuterVolumeSpecName: "console-config") pod "83dac517-6dc0-4404-b8e5-77b11d91cd9a" (UID: "83dac517-6dc0-4404-b8e5-77b11d91cd9a"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:55:57.555124 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:57.555107    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83dac517-6dc0-4404-b8e5-77b11d91cd9a-service-ca" (OuterVolumeSpecName: "service-ca") pod "83dac517-6dc0-4404-b8e5-77b11d91cd9a" (UID: "83dac517-6dc0-4404-b8e5-77b11d91cd9a"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:55:57.555124 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:57.555112    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83dac517-6dc0-4404-b8e5-77b11d91cd9a-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "83dac517-6dc0-4404-b8e5-77b11d91cd9a" (UID: "83dac517-6dc0-4404-b8e5-77b11d91cd9a"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:55:57.556914 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:57.556883    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83dac517-6dc0-4404-b8e5-77b11d91cd9a-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "83dac517-6dc0-4404-b8e5-77b11d91cd9a" (UID: "83dac517-6dc0-4404-b8e5-77b11d91cd9a"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:55:57.557013 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:57.556920    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83dac517-6dc0-4404-b8e5-77b11d91cd9a-kube-api-access-pcddq" (OuterVolumeSpecName: "kube-api-access-pcddq") pod "83dac517-6dc0-4404-b8e5-77b11d91cd9a" (UID: "83dac517-6dc0-4404-b8e5-77b11d91cd9a"). InnerVolumeSpecName "kube-api-access-pcddq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:55:57.557075 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:57.557004    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83dac517-6dc0-4404-b8e5-77b11d91cd9a-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "83dac517-6dc0-4404-b8e5-77b11d91cd9a" (UID: "83dac517-6dc0-4404-b8e5-77b11d91cd9a"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:55:57.655771 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:57.655727    2579 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/83dac517-6dc0-4404-b8e5-77b11d91cd9a-service-ca\") on node \"ip-10-0-138-224.ec2.internal\" DevicePath \"\""
Sep 09 20:55:57.655771 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:57.655770    2579 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/83dac517-6dc0-4404-b8e5-77b11d91cd9a-oauth-serving-cert\") on node \"ip-10-0-138-224.ec2.internal\" DevicePath \"\""
Sep 09 20:55:57.655923 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:57.655785    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pcddq\" (UniqueName: \"kubernetes.io/projected/83dac517-6dc0-4404-b8e5-77b11d91cd9a-kube-api-access-pcddq\") on node \"ip-10-0-138-224.ec2.internal\" DevicePath \"\""
Sep 09 20:55:57.655923 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:57.655799    2579 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/83dac517-6dc0-4404-b8e5-77b11d91cd9a-console-oauth-config\") on node \"ip-10-0-138-224.ec2.internal\" DevicePath \"\""
Sep 09 20:55:57.655923 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:57.655813    2579 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/83dac517-6dc0-4404-b8e5-77b11d91cd9a-console-serving-cert\") on node \"ip-10-0-138-224.ec2.internal\" DevicePath \"\""
Sep 09 20:55:57.655923 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:57.655826    2579 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/83dac517-6dc0-4404-b8e5-77b11d91cd9a-trusted-ca-bundle\") on node \"ip-10-0-138-224.ec2.internal\" DevicePath \"\""
Sep 09 20:55:57.655923 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:57.655837    2579 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/83dac517-6dc0-4404-b8e5-77b11d91cd9a-console-config\") on node \"ip-10-0-138-224.ec2.internal\" DevicePath \"\""
Sep 09 20:55:58.226617 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:58.226584    2579 generic.go:356] "Generic (PLEG): container finished" podID="83dac517-6dc0-4404-b8e5-77b11d91cd9a" containerID="2b9291d518c957dc3e5384cd8720fd8c6f394587b52e5bf826ecdd9bc6a978d9" exitCode=0
Sep 09 20:55:58.227012 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:58.226627    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5495fd7677-lg774"
Sep 09 20:55:58.227012 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:58.226636    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5495fd7677-lg774" event={"ID":"83dac517-6dc0-4404-b8e5-77b11d91cd9a","Type":"ContainerDied","Data":"2b9291d518c957dc3e5384cd8720fd8c6f394587b52e5bf826ecdd9bc6a978d9"}
Sep 09 20:55:58.227012 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:58.226682    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5495fd7677-lg774" event={"ID":"83dac517-6dc0-4404-b8e5-77b11d91cd9a","Type":"ContainerDied","Data":"e6924229be342b678c6e4bba52ad3988d53f8c176cb12a82151e6d88b4d3a37e"}
Sep 09 20:55:58.227012 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:58.226704    2579 scope.go:117] "RemoveContainer" containerID="2b9291d518c957dc3e5384cd8720fd8c6f394587b52e5bf826ecdd9bc6a978d9"
Sep 09 20:55:58.235112 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:58.235096    2579 scope.go:117] "RemoveContainer" containerID="2b9291d518c957dc3e5384cd8720fd8c6f394587b52e5bf826ecdd9bc6a978d9"
Sep 09 20:55:58.235383 ip-10-0-138-224 kubenswrapper[2579]: E0909 20:55:58.235365    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b9291d518c957dc3e5384cd8720fd8c6f394587b52e5bf826ecdd9bc6a978d9\": container with ID starting with 2b9291d518c957dc3e5384cd8720fd8c6f394587b52e5bf826ecdd9bc6a978d9 not found: ID does not exist" containerID="2b9291d518c957dc3e5384cd8720fd8c6f394587b52e5bf826ecdd9bc6a978d9"
Sep 09 20:55:58.235420 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:58.235395    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b9291d518c957dc3e5384cd8720fd8c6f394587b52e5bf826ecdd9bc6a978d9"} err="failed to get container status \"2b9291d518c957dc3e5384cd8720fd8c6f394587b52e5bf826ecdd9bc6a978d9\": rpc error: code = NotFound desc = could not find container \"2b9291d518c957dc3e5384cd8720fd8c6f394587b52e5bf826ecdd9bc6a978d9\": container with ID starting with 2b9291d518c957dc3e5384cd8720fd8c6f394587b52e5bf826ecdd9bc6a978d9 not found: ID does not exist"
Sep 09 20:55:58.248129 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:58.248104    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5495fd7677-lg774"]
Sep 09 20:55:58.251781 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:58.251752    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5495fd7677-lg774"]
Sep 09 20:55:59.290214 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:55:59.290168    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="83dac517-6dc0-4404-b8e5-77b11d91cd9a" path="/var/lib/kubelet/pods/83dac517-6dc0-4404-b8e5-77b11d91cd9a/volumes"
Sep 09 20:56:33.905095 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:56:33.905055    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6cb67bc6fb-ndjcw"]
Sep 09 20:56:33.905589 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:56:33.905398    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83dac517-6dc0-4404-b8e5-77b11d91cd9a" containerName="console"
Sep 09 20:56:33.905589 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:56:33.905411    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="83dac517-6dc0-4404-b8e5-77b11d91cd9a" containerName="console"
Sep 09 20:56:33.905589 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:56:33.905486    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="83dac517-6dc0-4404-b8e5-77b11d91cd9a" containerName="console"
Sep 09 20:56:33.909662 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:56:33.909647    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6cb67bc6fb-ndjcw"
Sep 09 20:56:33.921596 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:56:33.921558    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6cb67bc6fb-ndjcw"]
Sep 09 20:56:33.951784 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:56:33.951758    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/43dd5d9e-bcb8-4ab8-976e-ff100f6bdd1c-oauth-serving-cert\") pod \"console-6cb67bc6fb-ndjcw\" (UID: \"43dd5d9e-bcb8-4ab8-976e-ff100f6bdd1c\") " pod="openshift-console/console-6cb67bc6fb-ndjcw"
Sep 09 20:56:33.951921 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:56:33.951806    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvwq4\" (UniqueName: \"kubernetes.io/projected/43dd5d9e-bcb8-4ab8-976e-ff100f6bdd1c-kube-api-access-gvwq4\") pod \"console-6cb67bc6fb-ndjcw\" (UID: \"43dd5d9e-bcb8-4ab8-976e-ff100f6bdd1c\") " pod="openshift-console/console-6cb67bc6fb-ndjcw"
Sep 09 20:56:33.951921 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:56:33.951827    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/43dd5d9e-bcb8-4ab8-976e-ff100f6bdd1c-console-oauth-config\") pod \"console-6cb67bc6fb-ndjcw\" (UID: \"43dd5d9e-bcb8-4ab8-976e-ff100f6bdd1c\") " pod="openshift-console/console-6cb67bc6fb-ndjcw"
Sep 09 20:56:33.951921 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:56:33.951898    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/43dd5d9e-bcb8-4ab8-976e-ff100f6bdd1c-console-config\") pod \"console-6cb67bc6fb-ndjcw\" (UID: \"43dd5d9e-bcb8-4ab8-976e-ff100f6bdd1c\") " pod="openshift-console/console-6cb67bc6fb-ndjcw"
Sep 09 20:56:33.952022 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:56:33.951951    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/43dd5d9e-bcb8-4ab8-976e-ff100f6bdd1c-console-serving-cert\") pod \"console-6cb67bc6fb-ndjcw\" (UID: \"43dd5d9e-bcb8-4ab8-976e-ff100f6bdd1c\") " pod="openshift-console/console-6cb67bc6fb-ndjcw"
Sep 09 20:56:33.952022 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:56:33.951978    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/43dd5d9e-bcb8-4ab8-976e-ff100f6bdd1c-trusted-ca-bundle\") pod \"console-6cb67bc6fb-ndjcw\" (UID: \"43dd5d9e-bcb8-4ab8-976e-ff100f6bdd1c\") " pod="openshift-console/console-6cb67bc6fb-ndjcw"
Sep 09 20:56:33.952022 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:56:33.952015    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/43dd5d9e-bcb8-4ab8-976e-ff100f6bdd1c-service-ca\") pod \"console-6cb67bc6fb-ndjcw\" (UID: \"43dd5d9e-bcb8-4ab8-976e-ff100f6bdd1c\") " pod="openshift-console/console-6cb67bc6fb-ndjcw"
Sep 09 20:56:34.053029 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:56:34.052999    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvwq4\" (UniqueName: \"kubernetes.io/projected/43dd5d9e-bcb8-4ab8-976e-ff100f6bdd1c-kube-api-access-gvwq4\") pod \"console-6cb67bc6fb-ndjcw\" (UID: \"43dd5d9e-bcb8-4ab8-976e-ff100f6bdd1c\") " pod="openshift-console/console-6cb67bc6fb-ndjcw"
Sep 09 20:56:34.053162 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:56:34.053035    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/43dd5d9e-bcb8-4ab8-976e-ff100f6bdd1c-console-oauth-config\") pod \"console-6cb67bc6fb-ndjcw\" (UID: \"43dd5d9e-bcb8-4ab8-976e-ff100f6bdd1c\") " pod="openshift-console/console-6cb67bc6fb-ndjcw"
Sep 09 20:56:34.053162 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:56:34.053063    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/43dd5d9e-bcb8-4ab8-976e-ff100f6bdd1c-console-config\") pod \"console-6cb67bc6fb-ndjcw\" (UID: \"43dd5d9e-bcb8-4ab8-976e-ff100f6bdd1c\") " pod="openshift-console/console-6cb67bc6fb-ndjcw"
Sep 09 20:56:34.053162 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:56:34.053096    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/43dd5d9e-bcb8-4ab8-976e-ff100f6bdd1c-console-serving-cert\") pod \"console-6cb67bc6fb-ndjcw\" (UID: \"43dd5d9e-bcb8-4ab8-976e-ff100f6bdd1c\") " pod="openshift-console/console-6cb67bc6fb-ndjcw"
Sep 09 20:56:34.053162 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:56:34.053119    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/43dd5d9e-bcb8-4ab8-976e-ff100f6bdd1c-trusted-ca-bundle\") pod \"console-6cb67bc6fb-ndjcw\" (UID: \"43dd5d9e-bcb8-4ab8-976e-ff100f6bdd1c\") " pod="openshift-console/console-6cb67bc6fb-ndjcw"
Sep 09 20:56:34.053162 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:56:34.053137    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/43dd5d9e-bcb8-4ab8-976e-ff100f6bdd1c-service-ca\") pod \"console-6cb67bc6fb-ndjcw\" (UID: \"43dd5d9e-bcb8-4ab8-976e-ff100f6bdd1c\") " pod="openshift-console/console-6cb67bc6fb-ndjcw"
Sep 09 20:56:34.053384 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:56:34.053164    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/43dd5d9e-bcb8-4ab8-976e-ff100f6bdd1c-oauth-serving-cert\") pod \"console-6cb67bc6fb-ndjcw\" (UID: \"43dd5d9e-bcb8-4ab8-976e-ff100f6bdd1c\") " pod="openshift-console/console-6cb67bc6fb-ndjcw"
Sep 09 20:56:34.054020 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:56:34.053910    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/43dd5d9e-bcb8-4ab8-976e-ff100f6bdd1c-oauth-serving-cert\") pod \"console-6cb67bc6fb-ndjcw\" (UID: \"43dd5d9e-bcb8-4ab8-976e-ff100f6bdd1c\") " pod="openshift-console/console-6cb67bc6fb-ndjcw"
Sep 09 20:56:34.054020 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:56:34.053979    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/43dd5d9e-bcb8-4ab8-976e-ff100f6bdd1c-service-ca\") pod \"console-6cb67bc6fb-ndjcw\" (UID: \"43dd5d9e-bcb8-4ab8-976e-ff100f6bdd1c\") " pod="openshift-console/console-6cb67bc6fb-ndjcw"
Sep 09 20:56:34.054209 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:56:34.054030    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/43dd5d9e-bcb8-4ab8-976e-ff100f6bdd1c-console-config\") pod \"console-6cb67bc6fb-ndjcw\" (UID: \"43dd5d9e-bcb8-4ab8-976e-ff100f6bdd1c\") " pod="openshift-console/console-6cb67bc6fb-ndjcw"
Sep 09 20:56:34.054269 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:56:34.054218    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/43dd5d9e-bcb8-4ab8-976e-ff100f6bdd1c-trusted-ca-bundle\") pod \"console-6cb67bc6fb-ndjcw\" (UID: \"43dd5d9e-bcb8-4ab8-976e-ff100f6bdd1c\") " pod="openshift-console/console-6cb67bc6fb-ndjcw"
Sep 09 20:56:34.055720 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:56:34.055698    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/43dd5d9e-bcb8-4ab8-976e-ff100f6bdd1c-console-serving-cert\") pod \"console-6cb67bc6fb-ndjcw\" (UID: \"43dd5d9e-bcb8-4ab8-976e-ff100f6bdd1c\") " pod="openshift-console/console-6cb67bc6fb-ndjcw"
Sep 09 20:56:34.055822 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:56:34.055741    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/43dd5d9e-bcb8-4ab8-976e-ff100f6bdd1c-console-oauth-config\") pod \"console-6cb67bc6fb-ndjcw\" (UID: \"43dd5d9e-bcb8-4ab8-976e-ff100f6bdd1c\") " pod="openshift-console/console-6cb67bc6fb-ndjcw"
Sep 09 20:56:34.061028 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:56:34.061007    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvwq4\" (UniqueName: \"kubernetes.io/projected/43dd5d9e-bcb8-4ab8-976e-ff100f6bdd1c-kube-api-access-gvwq4\") pod \"console-6cb67bc6fb-ndjcw\" (UID: \"43dd5d9e-bcb8-4ab8-976e-ff100f6bdd1c\") " pod="openshift-console/console-6cb67bc6fb-ndjcw"
Sep 09 20:56:34.218978 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:56:34.218933    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6cb67bc6fb-ndjcw"
Sep 09 20:56:34.494041 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:56:34.493949    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6cb67bc6fb-ndjcw"]
Sep 09 20:56:34.496902 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:56:34.496876    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43dd5d9e_bcb8_4ab8_976e_ff100f6bdd1c.slice/crio-e361fe14bf4a6355411bff883a5ce2cc48ab72af4c564125f8454abc0a9e7602 WatchSource:0}: Error finding container e361fe14bf4a6355411bff883a5ce2cc48ab72af4c564125f8454abc0a9e7602: Status 404 returned error can't find the container with id e361fe14bf4a6355411bff883a5ce2cc48ab72af4c564125f8454abc0a9e7602
Sep 09 20:56:35.373281 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:56:35.373243    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6cb67bc6fb-ndjcw" event={"ID":"43dd5d9e-bcb8-4ab8-976e-ff100f6bdd1c","Type":"ContainerStarted","Data":"834070a392742738bc6703eea018b5ce076b1950f55006faaadde0843390604b"}
Sep 09 20:56:35.373780 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:56:35.373306    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6cb67bc6fb-ndjcw" event={"ID":"43dd5d9e-bcb8-4ab8-976e-ff100f6bdd1c","Type":"ContainerStarted","Data":"e361fe14bf4a6355411bff883a5ce2cc48ab72af4c564125f8454abc0a9e7602"}
Sep 09 20:56:35.393189 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:56:35.393143    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6cb67bc6fb-ndjcw" podStartSLOduration=2.393127119 podStartE2EDuration="2.393127119s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:56:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:56:35.392589022 +0000 UTC m=+252.632683180" watchObservedRunningTime="2026-09-09 20:56:35.393127119 +0000 UTC m=+252.633221249"
Sep 09 20:56:44.220147 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:56:44.220107    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6cb67bc6fb-ndjcw"
Sep 09 20:56:44.220644 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:56:44.220172    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6cb67bc6fb-ndjcw"
Sep 09 20:56:44.224775 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:56:44.224754    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6cb67bc6fb-ndjcw"
Sep 09 20:56:44.410499 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:56:44.410411    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6cb67bc6fb-ndjcw"
Sep 09 20:56:44.489884 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:56:44.489805    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-549b6ccc54-7n8jr"]
Sep 09 20:57:09.509760 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:57:09.509688    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-549b6ccc54-7n8jr" podUID="eb40731c-836a-454f-9dad-af5808c13ba8" containerName="console" containerID="cri-o://69d1043fe71368c7dfa1baf806fa2ae753b250902d34f79dad8984d33661a183" gracePeriod=15
Sep 09 20:57:09.754294 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:57:09.754266    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-549b6ccc54-7n8jr"
Sep 09 20:57:09.853159 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:57:09.853060    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/eb40731c-836a-454f-9dad-af5808c13ba8-console-oauth-config\") pod \"eb40731c-836a-454f-9dad-af5808c13ba8\" (UID: \"eb40731c-836a-454f-9dad-af5808c13ba8\") "
Sep 09 20:57:09.853159 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:57:09.853140    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/eb40731c-836a-454f-9dad-af5808c13ba8-oauth-serving-cert\") pod \"eb40731c-836a-454f-9dad-af5808c13ba8\" (UID: \"eb40731c-836a-454f-9dad-af5808c13ba8\") "
Sep 09 20:57:09.853386 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:57:09.853165    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/eb40731c-836a-454f-9dad-af5808c13ba8-console-config\") pod \"eb40731c-836a-454f-9dad-af5808c13ba8\" (UID: \"eb40731c-836a-454f-9dad-af5808c13ba8\") "
Sep 09 20:57:09.853386 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:57:09.853187    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/eb40731c-836a-454f-9dad-af5808c13ba8-service-ca\") pod \"eb40731c-836a-454f-9dad-af5808c13ba8\" (UID: \"eb40731c-836a-454f-9dad-af5808c13ba8\") "
Sep 09 20:57:09.853386 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:57:09.853211    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/eb40731c-836a-454f-9dad-af5808c13ba8-console-serving-cert\") pod \"eb40731c-836a-454f-9dad-af5808c13ba8\" (UID: \"eb40731c-836a-454f-9dad-af5808c13ba8\") "
Sep 09 20:57:09.853386 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:57:09.853232    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sh642\" (UniqueName: \"kubernetes.io/projected/eb40731c-836a-454f-9dad-af5808c13ba8-kube-api-access-sh642\") pod \"eb40731c-836a-454f-9dad-af5808c13ba8\" (UID: \"eb40731c-836a-454f-9dad-af5808c13ba8\") "
Sep 09 20:57:09.853386 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:57:09.853272    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eb40731c-836a-454f-9dad-af5808c13ba8-trusted-ca-bundle\") pod \"eb40731c-836a-454f-9dad-af5808c13ba8\" (UID: \"eb40731c-836a-454f-9dad-af5808c13ba8\") "
Sep 09 20:57:09.853699 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:57:09.853669    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb40731c-836a-454f-9dad-af5808c13ba8-console-config" (OuterVolumeSpecName: "console-config") pod "eb40731c-836a-454f-9dad-af5808c13ba8" (UID: "eb40731c-836a-454f-9dad-af5808c13ba8"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:57:09.853767 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:57:09.853673    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb40731c-836a-454f-9dad-af5808c13ba8-service-ca" (OuterVolumeSpecName: "service-ca") pod "eb40731c-836a-454f-9dad-af5808c13ba8" (UID: "eb40731c-836a-454f-9dad-af5808c13ba8"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:57:09.853884 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:57:09.853860    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb40731c-836a-454f-9dad-af5808c13ba8-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "eb40731c-836a-454f-9dad-af5808c13ba8" (UID: "eb40731c-836a-454f-9dad-af5808c13ba8"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:57:09.853884 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:57:09.853887    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb40731c-836a-454f-9dad-af5808c13ba8-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "eb40731c-836a-454f-9dad-af5808c13ba8" (UID: "eb40731c-836a-454f-9dad-af5808c13ba8"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:57:09.855474 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:57:09.855432    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb40731c-836a-454f-9dad-af5808c13ba8-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "eb40731c-836a-454f-9dad-af5808c13ba8" (UID: "eb40731c-836a-454f-9dad-af5808c13ba8"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:57:09.856042 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:57:09.856017    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb40731c-836a-454f-9dad-af5808c13ba8-kube-api-access-sh642" (OuterVolumeSpecName: "kube-api-access-sh642") pod "eb40731c-836a-454f-9dad-af5808c13ba8" (UID: "eb40731c-836a-454f-9dad-af5808c13ba8"). InnerVolumeSpecName "kube-api-access-sh642". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:57:09.856114 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:57:09.856014    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb40731c-836a-454f-9dad-af5808c13ba8-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "eb40731c-836a-454f-9dad-af5808c13ba8" (UID: "eb40731c-836a-454f-9dad-af5808c13ba8"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:57:09.954602 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:57:09.954552    2579 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/eb40731c-836a-454f-9dad-af5808c13ba8-console-serving-cert\") on node \"ip-10-0-138-224.ec2.internal\" DevicePath \"\""
Sep 09 20:57:09.954602 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:57:09.954594    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sh642\" (UniqueName: \"kubernetes.io/projected/eb40731c-836a-454f-9dad-af5808c13ba8-kube-api-access-sh642\") on node \"ip-10-0-138-224.ec2.internal\" DevicePath \"\""
Sep 09 20:57:09.954602 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:57:09.954605    2579 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eb40731c-836a-454f-9dad-af5808c13ba8-trusted-ca-bundle\") on node \"ip-10-0-138-224.ec2.internal\" DevicePath \"\""
Sep 09 20:57:09.954602 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:57:09.954613    2579 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/eb40731c-836a-454f-9dad-af5808c13ba8-console-oauth-config\") on node \"ip-10-0-138-224.ec2.internal\" DevicePath \"\""
Sep 09 20:57:09.954602 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:57:09.954621    2579 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/eb40731c-836a-454f-9dad-af5808c13ba8-oauth-serving-cert\") on node \"ip-10-0-138-224.ec2.internal\" DevicePath \"\""
Sep 09 20:57:09.954886 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:57:09.954631    2579 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/eb40731c-836a-454f-9dad-af5808c13ba8-console-config\") on node \"ip-10-0-138-224.ec2.internal\" DevicePath \"\""
Sep 09 20:57:09.954886 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:57:09.954639    2579 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/eb40731c-836a-454f-9dad-af5808c13ba8-service-ca\") on node \"ip-10-0-138-224.ec2.internal\" DevicePath \"\""
Sep 09 20:57:10.505771 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:57:10.505734    2579 generic.go:356] "Generic (PLEG): container finished" podID="eb40731c-836a-454f-9dad-af5808c13ba8" containerID="69d1043fe71368c7dfa1baf806fa2ae753b250902d34f79dad8984d33661a183" exitCode=0
Sep 09 20:57:10.505961 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:57:10.505796    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-549b6ccc54-7n8jr"
Sep 09 20:57:10.505961 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:57:10.505801    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-549b6ccc54-7n8jr" event={"ID":"eb40731c-836a-454f-9dad-af5808c13ba8","Type":"ContainerDied","Data":"69d1043fe71368c7dfa1baf806fa2ae753b250902d34f79dad8984d33661a183"}
Sep 09 20:57:10.505961 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:57:10.505935    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-549b6ccc54-7n8jr" event={"ID":"eb40731c-836a-454f-9dad-af5808c13ba8","Type":"ContainerDied","Data":"a994be64959fb3984dc9fca66cbd332829da101917f942fd771f1acd9bbbc263"}
Sep 09 20:57:10.506066 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:57:10.505965    2579 scope.go:117] "RemoveContainer" containerID="69d1043fe71368c7dfa1baf806fa2ae753b250902d34f79dad8984d33661a183"
Sep 09 20:57:10.515100 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:57:10.514923    2579 scope.go:117] "RemoveContainer" containerID="69d1043fe71368c7dfa1baf806fa2ae753b250902d34f79dad8984d33661a183"
Sep 09 20:57:10.515309 ip-10-0-138-224 kubenswrapper[2579]: E0909 20:57:10.515230    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69d1043fe71368c7dfa1baf806fa2ae753b250902d34f79dad8984d33661a183\": container with ID starting with 69d1043fe71368c7dfa1baf806fa2ae753b250902d34f79dad8984d33661a183 not found: ID does not exist" containerID="69d1043fe71368c7dfa1baf806fa2ae753b250902d34f79dad8984d33661a183"
Sep 09 20:57:10.515309 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:57:10.515256    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69d1043fe71368c7dfa1baf806fa2ae753b250902d34f79dad8984d33661a183"} err="failed to get container status \"69d1043fe71368c7dfa1baf806fa2ae753b250902d34f79dad8984d33661a183\": rpc error: code = NotFound desc = could not find container \"69d1043fe71368c7dfa1baf806fa2ae753b250902d34f79dad8984d33661a183\": container with ID starting with 69d1043fe71368c7dfa1baf806fa2ae753b250902d34f79dad8984d33661a183 not found: ID does not exist"
Sep 09 20:57:10.526266 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:57:10.526237    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-549b6ccc54-7n8jr"]
Sep 09 20:57:10.530625 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:57:10.530603    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-549b6ccc54-7n8jr"]
Sep 09 20:57:11.290635 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:57:11.290602    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eb40731c-836a-454f-9dad-af5808c13ba8" path="/var/lib/kubelet/pods/eb40731c-836a-454f-9dad-af5808c13ba8/volumes"
Sep 09 20:57:23.170620 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:57:23.170594    2579 kubelet.go:1648] "Image garbage collection succeeded"
Sep 09 20:58:16.664939 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:58:16.664867    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-grsvc"]
Sep 09 20:58:16.665318 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:58:16.665164    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb40731c-836a-454f-9dad-af5808c13ba8" containerName="console"
Sep 09 20:58:16.665318 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:58:16.665177    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb40731c-836a-454f-9dad-af5808c13ba8" containerName="console"
Sep 09 20:58:16.665318 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:58:16.665237    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="eb40731c-836a-454f-9dad-af5808c13ba8" containerName="console"
Sep 09 20:58:16.668041 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:58:16.668025    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-grsvc"
Sep 09 20:58:16.671981 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:58:16.671959    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
Sep 09 20:58:16.672261 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:58:16.672239    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Sep 09 20:58:16.672332 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:58:16.672242    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Sep 09 20:58:16.672370 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:58:16.672332    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"custom-metrics-autoscaler-operator-dockercfg-fswh6\""
Sep 09 20:58:16.729296 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:58:16.729268    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-grsvc"]
Sep 09 20:58:16.751142 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:58:16.751111    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fzdql\" (UniqueName: \"kubernetes.io/projected/3643738e-2617-4dae-affe-95be80725476-kube-api-access-fzdql\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-grsvc\" (UID: \"3643738e-2617-4dae-affe-95be80725476\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-grsvc"
Sep 09 20:58:16.751253 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:58:16.751197    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/3643738e-2617-4dae-affe-95be80725476-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-grsvc\" (UID: \"3643738e-2617-4dae-affe-95be80725476\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-grsvc"
Sep 09 20:58:16.851687 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:58:16.851657    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/3643738e-2617-4dae-affe-95be80725476-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-grsvc\" (UID: \"3643738e-2617-4dae-affe-95be80725476\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-grsvc"
Sep 09 20:58:16.851828 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:58:16.851707    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzdql\" (UniqueName: \"kubernetes.io/projected/3643738e-2617-4dae-affe-95be80725476-kube-api-access-fzdql\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-grsvc\" (UID: \"3643738e-2617-4dae-affe-95be80725476\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-grsvc"
Sep 09 20:58:16.854145 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:58:16.854122    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/3643738e-2617-4dae-affe-95be80725476-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-grsvc\" (UID: \"3643738e-2617-4dae-affe-95be80725476\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-grsvc"
Sep 09 20:58:16.874916 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:58:16.874890    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzdql\" (UniqueName: \"kubernetes.io/projected/3643738e-2617-4dae-affe-95be80725476-kube-api-access-fzdql\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-grsvc\" (UID: \"3643738e-2617-4dae-affe-95be80725476\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-grsvc"
Sep 09 20:58:16.977339 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:58:16.977263    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-grsvc"
Sep 09 20:58:17.288602 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:58:17.288572    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3643738e_2617_4dae_affe_95be80725476.slice/crio-6c88de3247c081ae3674526f33c5ad488116fed28f33d756ad3a89019e591fab WatchSource:0}: Error finding container 6c88de3247c081ae3674526f33c5ad488116fed28f33d756ad3a89019e591fab: Status 404 returned error can't find the container with id 6c88de3247c081ae3674526f33c5ad488116fed28f33d756ad3a89019e591fab
Sep 09 20:58:17.290536 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:58:17.290515    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 20:58:17.290847 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:58:17.290824    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-grsvc"]
Sep 09 20:58:17.769730 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:58:17.769693    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-grsvc" event={"ID":"3643738e-2617-4dae-affe-95be80725476","Type":"ContainerStarted","Data":"6c88de3247c081ae3674526f33c5ad488116fed28f33d756ad3a89019e591fab"}
Sep 09 20:58:20.785377 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:58:20.785283    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-grsvc" event={"ID":"3643738e-2617-4dae-affe-95be80725476","Type":"ContainerStarted","Data":"3fba148257143c29b9f255e7ab4deca3b00108b6694a662f7bef74e780c4e7a5"}
Sep 09 20:58:20.785736 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:58:20.785395    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-grsvc"
Sep 09 20:58:41.792279 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:58:41.792248    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-grsvc"
Sep 09 20:58:41.811626 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:58:41.811574    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-grsvc" podStartSLOduration=22.586383649 podStartE2EDuration="25.811554497s" totalImagesPullingTime="3.225170848s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:58:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:58:20.824267273 +0000 UTC m=+358.064361391" watchObservedRunningTime="2026-09-09 20:58:41.811554497 +0000 UTC m=+379.051648614"
Sep 09 20:58:48.405879 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:58:48.405847    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5c694d5bb7-6q8xg"]
Sep 09 20:58:48.409024 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:58:48.409006    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5c694d5bb7-6q8xg"
Sep 09 20:58:48.419420 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:58:48.419392    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5c694d5bb7-6q8xg"]
Sep 09 20:58:48.492529 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:58:48.492491    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/746817b6-f0bd-4b14-b04e-4c30116d0045-trusted-ca-bundle\") pod \"console-5c694d5bb7-6q8xg\" (UID: \"746817b6-f0bd-4b14-b04e-4c30116d0045\") " pod="openshift-console/console-5c694d5bb7-6q8xg"
Sep 09 20:58:48.492645 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:58:48.492543    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g54f9\" (UniqueName: \"kubernetes.io/projected/746817b6-f0bd-4b14-b04e-4c30116d0045-kube-api-access-g54f9\") pod \"console-5c694d5bb7-6q8xg\" (UID: \"746817b6-f0bd-4b14-b04e-4c30116d0045\") " pod="openshift-console/console-5c694d5bb7-6q8xg"
Sep 09 20:58:48.492645 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:58:48.492566    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/746817b6-f0bd-4b14-b04e-4c30116d0045-service-ca\") pod \"console-5c694d5bb7-6q8xg\" (UID: \"746817b6-f0bd-4b14-b04e-4c30116d0045\") " pod="openshift-console/console-5c694d5bb7-6q8xg"
Sep 09 20:58:48.492645 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:58:48.492596    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/746817b6-f0bd-4b14-b04e-4c30116d0045-console-serving-cert\") pod \"console-5c694d5bb7-6q8xg\" (UID: \"746817b6-f0bd-4b14-b04e-4c30116d0045\") " pod="openshift-console/console-5c694d5bb7-6q8xg"
Sep 09 20:58:48.492645 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:58:48.492611    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/746817b6-f0bd-4b14-b04e-4c30116d0045-console-config\") pod \"console-5c694d5bb7-6q8xg\" (UID: \"746817b6-f0bd-4b14-b04e-4c30116d0045\") " pod="openshift-console/console-5c694d5bb7-6q8xg"
Sep 09 20:58:48.492645 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:58:48.492625    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/746817b6-f0bd-4b14-b04e-4c30116d0045-oauth-serving-cert\") pod \"console-5c694d5bb7-6q8xg\" (UID: \"746817b6-f0bd-4b14-b04e-4c30116d0045\") " pod="openshift-console/console-5c694d5bb7-6q8xg"
Sep 09 20:58:48.492815 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:58:48.492681    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/746817b6-f0bd-4b14-b04e-4c30116d0045-console-oauth-config\") pod \"console-5c694d5bb7-6q8xg\" (UID: \"746817b6-f0bd-4b14-b04e-4c30116d0045\") " pod="openshift-console/console-5c694d5bb7-6q8xg"
Sep 09 20:58:48.593807 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:58:48.593782    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/746817b6-f0bd-4b14-b04e-4c30116d0045-trusted-ca-bundle\") pod \"console-5c694d5bb7-6q8xg\" (UID: \"746817b6-f0bd-4b14-b04e-4c30116d0045\") " pod="openshift-console/console-5c694d5bb7-6q8xg"
Sep 09 20:58:48.593923 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:58:48.593820    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g54f9\" (UniqueName: \"kubernetes.io/projected/746817b6-f0bd-4b14-b04e-4c30116d0045-kube-api-access-g54f9\") pod \"console-5c694d5bb7-6q8xg\" (UID: \"746817b6-f0bd-4b14-b04e-4c30116d0045\") " pod="openshift-console/console-5c694d5bb7-6q8xg"
Sep 09 20:58:48.593923 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:58:48.593840    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/746817b6-f0bd-4b14-b04e-4c30116d0045-service-ca\") pod \"console-5c694d5bb7-6q8xg\" (UID: \"746817b6-f0bd-4b14-b04e-4c30116d0045\") " pod="openshift-console/console-5c694d5bb7-6q8xg"
Sep 09 20:58:48.593923 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:58:48.593865    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/746817b6-f0bd-4b14-b04e-4c30116d0045-console-serving-cert\") pod \"console-5c694d5bb7-6q8xg\" (UID: \"746817b6-f0bd-4b14-b04e-4c30116d0045\") " pod="openshift-console/console-5c694d5bb7-6q8xg"
Sep 09 20:58:48.593923 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:58:48.593897    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/746817b6-f0bd-4b14-b04e-4c30116d0045-console-config\") pod \"console-5c694d5bb7-6q8xg\" (UID: \"746817b6-f0bd-4b14-b04e-4c30116d0045\") " pod="openshift-console/console-5c694d5bb7-6q8xg"
Sep 09 20:58:48.593923 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:58:48.593912    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/746817b6-f0bd-4b14-b04e-4c30116d0045-oauth-serving-cert\") pod \"console-5c694d5bb7-6q8xg\" (UID: \"746817b6-f0bd-4b14-b04e-4c30116d0045\") " pod="openshift-console/console-5c694d5bb7-6q8xg"
Sep 09 20:58:48.594188 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:58:48.593959    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/746817b6-f0bd-4b14-b04e-4c30116d0045-console-oauth-config\") pod \"console-5c694d5bb7-6q8xg\" (UID: \"746817b6-f0bd-4b14-b04e-4c30116d0045\") " pod="openshift-console/console-5c694d5bb7-6q8xg"
Sep 09 20:58:48.594683 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:58:48.594655    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/746817b6-f0bd-4b14-b04e-4c30116d0045-console-config\") pod \"console-5c694d5bb7-6q8xg\" (UID: \"746817b6-f0bd-4b14-b04e-4c30116d0045\") " pod="openshift-console/console-5c694d5bb7-6q8xg"
Sep 09 20:58:48.594683 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:58:48.594671    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/746817b6-f0bd-4b14-b04e-4c30116d0045-oauth-serving-cert\") pod \"console-5c694d5bb7-6q8xg\" (UID: \"746817b6-f0bd-4b14-b04e-4c30116d0045\") " pod="openshift-console/console-5c694d5bb7-6q8xg"
Sep 09 20:58:48.594979 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:58:48.594955    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/746817b6-f0bd-4b14-b04e-4c30116d0045-service-ca\") pod \"console-5c694d5bb7-6q8xg\" (UID: \"746817b6-f0bd-4b14-b04e-4c30116d0045\") " pod="openshift-console/console-5c694d5bb7-6q8xg"
Sep 09 20:58:48.595019 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:58:48.594960    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/746817b6-f0bd-4b14-b04e-4c30116d0045-trusted-ca-bundle\") pod \"console-5c694d5bb7-6q8xg\" (UID: \"746817b6-f0bd-4b14-b04e-4c30116d0045\") " pod="openshift-console/console-5c694d5bb7-6q8xg"
Sep 09 20:58:48.596602 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:58:48.596580    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/746817b6-f0bd-4b14-b04e-4c30116d0045-console-serving-cert\") pod \"console-5c694d5bb7-6q8xg\" (UID: \"746817b6-f0bd-4b14-b04e-4c30116d0045\") " pod="openshift-console/console-5c694d5bb7-6q8xg"
Sep 09 20:58:48.596672 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:58:48.596658    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/746817b6-f0bd-4b14-b04e-4c30116d0045-console-oauth-config\") pod \"console-5c694d5bb7-6q8xg\" (UID: \"746817b6-f0bd-4b14-b04e-4c30116d0045\") " pod="openshift-console/console-5c694d5bb7-6q8xg"
Sep 09 20:58:48.603753 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:58:48.603735    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g54f9\" (UniqueName: \"kubernetes.io/projected/746817b6-f0bd-4b14-b04e-4c30116d0045-kube-api-access-g54f9\") pod \"console-5c694d5bb7-6q8xg\" (UID: \"746817b6-f0bd-4b14-b04e-4c30116d0045\") " pod="openshift-console/console-5c694d5bb7-6q8xg"
Sep 09 20:58:48.717572 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:58:48.717536    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5c694d5bb7-6q8xg"
Sep 09 20:58:48.990743 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:58:48.990720    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5c694d5bb7-6q8xg"]
Sep 09 20:58:48.993151 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:58:48.993124    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod746817b6_f0bd_4b14_b04e_4c30116d0045.slice/crio-06ff9f1f1f83e814828512085753e6c1b2c5eff7d0011e4c15c14df7f8f96a13 WatchSource:0}: Error finding container 06ff9f1f1f83e814828512085753e6c1b2c5eff7d0011e4c15c14df7f8f96a13: Status 404 returned error can't find the container with id 06ff9f1f1f83e814828512085753e6c1b2c5eff7d0011e4c15c14df7f8f96a13
Sep 09 20:58:49.906987 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:58:49.906952    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5c694d5bb7-6q8xg" event={"ID":"746817b6-f0bd-4b14-b04e-4c30116d0045","Type":"ContainerStarted","Data":"4a7e99906360b98d535479f6e6a59929814b652f1cba02322749723807852682"}
Sep 09 20:58:49.907356 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:58:49.907008    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5c694d5bb7-6q8xg" event={"ID":"746817b6-f0bd-4b14-b04e-4c30116d0045","Type":"ContainerStarted","Data":"06ff9f1f1f83e814828512085753e6c1b2c5eff7d0011e4c15c14df7f8f96a13"}
Sep 09 20:58:49.927178 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:58:49.927132    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5c694d5bb7-6q8xg" podStartSLOduration=1.9271176410000002 podStartE2EDuration="1.927117641s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:58:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:58:49.92537911 +0000 UTC m=+387.165473229" watchObservedRunningTime="2026-09-09 20:58:49.927117641 +0000 UTC m=+387.167211760"
Sep 09 20:58:58.717853 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:58:58.717810    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5c694d5bb7-6q8xg"
Sep 09 20:58:58.718280 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:58:58.717901    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5c694d5bb7-6q8xg"
Sep 09 20:58:58.722639 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:58:58.722617    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5c694d5bb7-6q8xg"
Sep 09 20:58:58.954375 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:58:58.954344    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5c694d5bb7-6q8xg"
Sep 09 20:58:59.026636 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:58:59.026553    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6cb67bc6fb-ndjcw"]
Sep 09 20:59:24.046225 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:59:24.046178    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6cb67bc6fb-ndjcw" podUID="43dd5d9e-bcb8-4ab8-976e-ff100f6bdd1c" containerName="console" containerID="cri-o://834070a392742738bc6703eea018b5ce076b1950f55006faaadde0843390604b" gracePeriod=15
Sep 09 20:59:24.282457 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:59:24.282410    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6cb67bc6fb-ndjcw"
Sep 09 20:59:24.394261 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:59:24.394196    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/43dd5d9e-bcb8-4ab8-976e-ff100f6bdd1c-console-config\") pod \"43dd5d9e-bcb8-4ab8-976e-ff100f6bdd1c\" (UID: \"43dd5d9e-bcb8-4ab8-976e-ff100f6bdd1c\") "
Sep 09 20:59:24.394261 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:59:24.394245    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gvwq4\" (UniqueName: \"kubernetes.io/projected/43dd5d9e-bcb8-4ab8-976e-ff100f6bdd1c-kube-api-access-gvwq4\") pod \"43dd5d9e-bcb8-4ab8-976e-ff100f6bdd1c\" (UID: \"43dd5d9e-bcb8-4ab8-976e-ff100f6bdd1c\") "
Sep 09 20:59:24.394413 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:59:24.394289    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/43dd5d9e-bcb8-4ab8-976e-ff100f6bdd1c-trusted-ca-bundle\") pod \"43dd5d9e-bcb8-4ab8-976e-ff100f6bdd1c\" (UID: \"43dd5d9e-bcb8-4ab8-976e-ff100f6bdd1c\") "
Sep 09 20:59:24.394413 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:59:24.394379    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/43dd5d9e-bcb8-4ab8-976e-ff100f6bdd1c-service-ca\") pod \"43dd5d9e-bcb8-4ab8-976e-ff100f6bdd1c\" (UID: \"43dd5d9e-bcb8-4ab8-976e-ff100f6bdd1c\") "
Sep 09 20:59:24.394535 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:59:24.394427    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/43dd5d9e-bcb8-4ab8-976e-ff100f6bdd1c-oauth-serving-cert\") pod \"43dd5d9e-bcb8-4ab8-976e-ff100f6bdd1c\" (UID: \"43dd5d9e-bcb8-4ab8-976e-ff100f6bdd1c\") "
Sep 09 20:59:24.394535 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:59:24.394509    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/43dd5d9e-bcb8-4ab8-976e-ff100f6bdd1c-console-oauth-config\") pod \"43dd5d9e-bcb8-4ab8-976e-ff100f6bdd1c\" (UID: \"43dd5d9e-bcb8-4ab8-976e-ff100f6bdd1c\") "
Sep 09 20:59:24.394641 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:59:24.394538    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/43dd5d9e-bcb8-4ab8-976e-ff100f6bdd1c-console-serving-cert\") pod \"43dd5d9e-bcb8-4ab8-976e-ff100f6bdd1c\" (UID: \"43dd5d9e-bcb8-4ab8-976e-ff100f6bdd1c\") "
Sep 09 20:59:24.394697 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:59:24.394662    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43dd5d9e-bcb8-4ab8-976e-ff100f6bdd1c-console-config" (OuterVolumeSpecName: "console-config") pod "43dd5d9e-bcb8-4ab8-976e-ff100f6bdd1c" (UID: "43dd5d9e-bcb8-4ab8-976e-ff100f6bdd1c"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:59:24.394825 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:59:24.394799    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43dd5d9e-bcb8-4ab8-976e-ff100f6bdd1c-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "43dd5d9e-bcb8-4ab8-976e-ff100f6bdd1c" (UID: "43dd5d9e-bcb8-4ab8-976e-ff100f6bdd1c"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:59:24.394825 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:59:24.394809    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43dd5d9e-bcb8-4ab8-976e-ff100f6bdd1c-service-ca" (OuterVolumeSpecName: "service-ca") pod "43dd5d9e-bcb8-4ab8-976e-ff100f6bdd1c" (UID: "43dd5d9e-bcb8-4ab8-976e-ff100f6bdd1c"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:59:24.394945 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:59:24.394825    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43dd5d9e-bcb8-4ab8-976e-ff100f6bdd1c-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "43dd5d9e-bcb8-4ab8-976e-ff100f6bdd1c" (UID: "43dd5d9e-bcb8-4ab8-976e-ff100f6bdd1c"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:59:24.394996 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:59:24.394941    2579 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/43dd5d9e-bcb8-4ab8-976e-ff100f6bdd1c-console-config\") on node \"ip-10-0-138-224.ec2.internal\" DevicePath \"\""
Sep 09 20:59:24.394996 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:59:24.394961    2579 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/43dd5d9e-bcb8-4ab8-976e-ff100f6bdd1c-trusted-ca-bundle\") on node \"ip-10-0-138-224.ec2.internal\" DevicePath \"\""
Sep 09 20:59:24.394996 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:59:24.394974    2579 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/43dd5d9e-bcb8-4ab8-976e-ff100f6bdd1c-service-ca\") on node \"ip-10-0-138-224.ec2.internal\" DevicePath \"\""
Sep 09 20:59:24.394996 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:59:24.394983    2579 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/43dd5d9e-bcb8-4ab8-976e-ff100f6bdd1c-oauth-serving-cert\") on node \"ip-10-0-138-224.ec2.internal\" DevicePath \"\""
Sep 09 20:59:24.396714 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:59:24.396690    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43dd5d9e-bcb8-4ab8-976e-ff100f6bdd1c-kube-api-access-gvwq4" (OuterVolumeSpecName: "kube-api-access-gvwq4") pod "43dd5d9e-bcb8-4ab8-976e-ff100f6bdd1c" (UID: "43dd5d9e-bcb8-4ab8-976e-ff100f6bdd1c"). InnerVolumeSpecName "kube-api-access-gvwq4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:59:24.396834 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:59:24.396817    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43dd5d9e-bcb8-4ab8-976e-ff100f6bdd1c-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "43dd5d9e-bcb8-4ab8-976e-ff100f6bdd1c" (UID: "43dd5d9e-bcb8-4ab8-976e-ff100f6bdd1c"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:59:24.396885 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:59:24.396867    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43dd5d9e-bcb8-4ab8-976e-ff100f6bdd1c-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "43dd5d9e-bcb8-4ab8-976e-ff100f6bdd1c" (UID: "43dd5d9e-bcb8-4ab8-976e-ff100f6bdd1c"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:59:24.496000 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:59:24.495970    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gvwq4\" (UniqueName: \"kubernetes.io/projected/43dd5d9e-bcb8-4ab8-976e-ff100f6bdd1c-kube-api-access-gvwq4\") on node \"ip-10-0-138-224.ec2.internal\" DevicePath \"\""
Sep 09 20:59:24.496000 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:59:24.495991    2579 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/43dd5d9e-bcb8-4ab8-976e-ff100f6bdd1c-console-oauth-config\") on node \"ip-10-0-138-224.ec2.internal\" DevicePath \"\""
Sep 09 20:59:24.496000 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:59:24.496001    2579 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/43dd5d9e-bcb8-4ab8-976e-ff100f6bdd1c-console-serving-cert\") on node \"ip-10-0-138-224.ec2.internal\" DevicePath \"\""
Sep 09 20:59:25.059195 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:59:25.059163    2579 generic.go:356] "Generic (PLEG): container finished" podID="43dd5d9e-bcb8-4ab8-976e-ff100f6bdd1c" containerID="834070a392742738bc6703eea018b5ce076b1950f55006faaadde0843390604b" exitCode=0
Sep 09 20:59:25.059614 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:59:25.059214    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6cb67bc6fb-ndjcw" event={"ID":"43dd5d9e-bcb8-4ab8-976e-ff100f6bdd1c","Type":"ContainerDied","Data":"834070a392742738bc6703eea018b5ce076b1950f55006faaadde0843390604b"}
Sep 09 20:59:25.059614 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:59:25.059273    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6cb67bc6fb-ndjcw" event={"ID":"43dd5d9e-bcb8-4ab8-976e-ff100f6bdd1c","Type":"ContainerDied","Data":"e361fe14bf4a6355411bff883a5ce2cc48ab72af4c564125f8454abc0a9e7602"}
Sep 09 20:59:25.059614 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:59:25.059216    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6cb67bc6fb-ndjcw"
Sep 09 20:59:25.059614 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:59:25.059293    2579 scope.go:117] "RemoveContainer" containerID="834070a392742738bc6703eea018b5ce076b1950f55006faaadde0843390604b"
Sep 09 20:59:25.068716 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:59:25.068698    2579 scope.go:117] "RemoveContainer" containerID="834070a392742738bc6703eea018b5ce076b1950f55006faaadde0843390604b"
Sep 09 20:59:25.068976 ip-10-0-138-224 kubenswrapper[2579]: E0909 20:59:25.068957    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"834070a392742738bc6703eea018b5ce076b1950f55006faaadde0843390604b\": container with ID starting with 834070a392742738bc6703eea018b5ce076b1950f55006faaadde0843390604b not found: ID does not exist" containerID="834070a392742738bc6703eea018b5ce076b1950f55006faaadde0843390604b"
Sep 09 20:59:25.069031 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:59:25.068989    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"834070a392742738bc6703eea018b5ce076b1950f55006faaadde0843390604b"} err="failed to get container status \"834070a392742738bc6703eea018b5ce076b1950f55006faaadde0843390604b\": rpc error: code = NotFound desc = could not find container \"834070a392742738bc6703eea018b5ce076b1950f55006faaadde0843390604b\": container with ID starting with 834070a392742738bc6703eea018b5ce076b1950f55006faaadde0843390604b not found: ID does not exist"
Sep 09 20:59:25.112991 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:59:25.112961    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6cb67bc6fb-ndjcw"]
Sep 09 20:59:25.115505 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:59:25.115485    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6cb67bc6fb-ndjcw"]
Sep 09 20:59:25.291200 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:59:25.291163    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="43dd5d9e-bcb8-4ab8-976e-ff100f6bdd1c" path="/var/lib/kubelet/pods/43dd5d9e-bcb8-4ab8-976e-ff100f6bdd1c/volumes"
Sep 09 20:59:45.973927 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:59:45.973856    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-644949856f-cg2jv"]
Sep 09 20:59:45.974238 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:59:45.974167    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43dd5d9e-bcb8-4ab8-976e-ff100f6bdd1c" containerName="console"
Sep 09 20:59:45.974238 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:59:45.974178    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="43dd5d9e-bcb8-4ab8-976e-ff100f6bdd1c" containerName="console"
Sep 09 20:59:45.974320 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:59:45.974241    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="43dd5d9e-bcb8-4ab8-976e-ff100f6bdd1c" containerName="console"
Sep 09 20:59:45.977033 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:59:45.977016    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-644949856f-cg2jv"
Sep 09 20:59:45.981230 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:59:45.981205    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-webhook-server-cert\""
Sep 09 20:59:45.981616 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:59:45.981596    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 09 20:59:45.981672 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:59:45.981604    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 09 20:59:45.981672 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:59:45.981604    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-controller-manager-dockercfg-xqhtd\""
Sep 09 20:59:45.990756 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:59:45.990736    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-644949856f-cg2jv"]
Sep 09 20:59:46.018465 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:59:46.018425    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-64568bcd49-jnjbr"]
Sep 09 20:59:46.021656 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:59:46.021639    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-jnjbr"
Sep 09 20:59:46.024127 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:59:46.024104    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Sep 09 20:59:46.024220 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:59:46.024183    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-jq25d\""
Sep 09 20:59:46.033702 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:59:46.033683    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-jnjbr"]
Sep 09 20:59:46.050734 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:59:46.050706    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dkdqq\" (UniqueName: \"kubernetes.io/projected/d1ab44fb-91bb-46a4-b592-5a7ef09bc86b-kube-api-access-dkdqq\") pod \"kserve-controller-manager-644949856f-cg2jv\" (UID: \"d1ab44fb-91bb-46a4-b592-5a7ef09bc86b\") " pod="kserve/kserve-controller-manager-644949856f-cg2jv"
Sep 09 20:59:46.050809 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:59:46.050752    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/6918bd2d-9c6b-48b3-acca-744ea4cae141-data\") pod \"seaweedfs-64568bcd49-jnjbr\" (UID: \"6918bd2d-9c6b-48b3-acca-744ea4cae141\") " pod="kserve/seaweedfs-64568bcd49-jnjbr"
Sep 09 20:59:46.050809 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:59:46.050772    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dspwn\" (UniqueName: \"kubernetes.io/projected/6918bd2d-9c6b-48b3-acca-744ea4cae141-kube-api-access-dspwn\") pod \"seaweedfs-64568bcd49-jnjbr\" (UID: \"6918bd2d-9c6b-48b3-acca-744ea4cae141\") " pod="kserve/seaweedfs-64568bcd49-jnjbr"
Sep 09 20:59:46.050890 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:59:46.050816    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d1ab44fb-91bb-46a4-b592-5a7ef09bc86b-cert\") pod \"kserve-controller-manager-644949856f-cg2jv\" (UID: \"d1ab44fb-91bb-46a4-b592-5a7ef09bc86b\") " pod="kserve/kserve-controller-manager-644949856f-cg2jv"
Sep 09 20:59:46.152216 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:59:46.152189    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkdqq\" (UniqueName: \"kubernetes.io/projected/d1ab44fb-91bb-46a4-b592-5a7ef09bc86b-kube-api-access-dkdqq\") pod \"kserve-controller-manager-644949856f-cg2jv\" (UID: \"d1ab44fb-91bb-46a4-b592-5a7ef09bc86b\") " pod="kserve/kserve-controller-manager-644949856f-cg2jv"
Sep 09 20:59:46.152336 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:59:46.152237    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/6918bd2d-9c6b-48b3-acca-744ea4cae141-data\") pod \"seaweedfs-64568bcd49-jnjbr\" (UID: \"6918bd2d-9c6b-48b3-acca-744ea4cae141\") " pod="kserve/seaweedfs-64568bcd49-jnjbr"
Sep 09 20:59:46.152336 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:59:46.152257    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dspwn\" (UniqueName: \"kubernetes.io/projected/6918bd2d-9c6b-48b3-acca-744ea4cae141-kube-api-access-dspwn\") pod \"seaweedfs-64568bcd49-jnjbr\" (UID: \"6918bd2d-9c6b-48b3-acca-744ea4cae141\") " pod="kserve/seaweedfs-64568bcd49-jnjbr"
Sep 09 20:59:46.152336 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:59:46.152277    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d1ab44fb-91bb-46a4-b592-5a7ef09bc86b-cert\") pod \"kserve-controller-manager-644949856f-cg2jv\" (UID: \"d1ab44fb-91bb-46a4-b592-5a7ef09bc86b\") " pod="kserve/kserve-controller-manager-644949856f-cg2jv"
Sep 09 20:59:46.152655 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:59:46.152631    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/6918bd2d-9c6b-48b3-acca-744ea4cae141-data\") pod \"seaweedfs-64568bcd49-jnjbr\" (UID: \"6918bd2d-9c6b-48b3-acca-744ea4cae141\") " pod="kserve/seaweedfs-64568bcd49-jnjbr"
Sep 09 20:59:46.154709 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:59:46.154689    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d1ab44fb-91bb-46a4-b592-5a7ef09bc86b-cert\") pod \"kserve-controller-manager-644949856f-cg2jv\" (UID: \"d1ab44fb-91bb-46a4-b592-5a7ef09bc86b\") " pod="kserve/kserve-controller-manager-644949856f-cg2jv"
Sep 09 20:59:46.161000 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:59:46.160982    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkdqq\" (UniqueName: \"kubernetes.io/projected/d1ab44fb-91bb-46a4-b592-5a7ef09bc86b-kube-api-access-dkdqq\") pod \"kserve-controller-manager-644949856f-cg2jv\" (UID: \"d1ab44fb-91bb-46a4-b592-5a7ef09bc86b\") " pod="kserve/kserve-controller-manager-644949856f-cg2jv"
Sep 09 20:59:46.161082 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:59:46.161060    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dspwn\" (UniqueName: \"kubernetes.io/projected/6918bd2d-9c6b-48b3-acca-744ea4cae141-kube-api-access-dspwn\") pod \"seaweedfs-64568bcd49-jnjbr\" (UID: \"6918bd2d-9c6b-48b3-acca-744ea4cae141\") " pod="kserve/seaweedfs-64568bcd49-jnjbr"
Sep 09 20:59:46.286498 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:59:46.286448    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-644949856f-cg2jv"
Sep 09 20:59:46.330573 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:59:46.330544    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-jnjbr"
Sep 09 20:59:46.558595 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:59:46.558527    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-644949856f-cg2jv"]
Sep 09 20:59:46.562054 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:59:46.562019    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1ab44fb_91bb_46a4_b592_5a7ef09bc86b.slice/crio-d116632a76ba2527cb26466d75c971e483445dab589c04dc5c1be21b91b42758 WatchSource:0}: Error finding container d116632a76ba2527cb26466d75c971e483445dab589c04dc5c1be21b91b42758: Status 404 returned error can't find the container with id d116632a76ba2527cb26466d75c971e483445dab589c04dc5c1be21b91b42758
Sep 09 20:59:46.607318 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:59:46.607294    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-jnjbr"]
Sep 09 20:59:46.610759 ip-10-0-138-224 kubenswrapper[2579]: W0909 20:59:46.610726    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6918bd2d_9c6b_48b3_acca_744ea4cae141.slice/crio-1a1d6a712abea5ee987fae27959e71fb2e1f3272dc57abc696f9a3d4e97ce19a WatchSource:0}: Error finding container 1a1d6a712abea5ee987fae27959e71fb2e1f3272dc57abc696f9a3d4e97ce19a: Status 404 returned error can't find the container with id 1a1d6a712abea5ee987fae27959e71fb2e1f3272dc57abc696f9a3d4e97ce19a
Sep 09 20:59:47.155106 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:59:47.155062    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-644949856f-cg2jv" event={"ID":"d1ab44fb-91bb-46a4-b592-5a7ef09bc86b","Type":"ContainerStarted","Data":"d116632a76ba2527cb26466d75c971e483445dab589c04dc5c1be21b91b42758"}
Sep 09 20:59:47.157062 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:59:47.157022    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-jnjbr" event={"ID":"6918bd2d-9c6b-48b3-acca-744ea4cae141","Type":"ContainerStarted","Data":"1a1d6a712abea5ee987fae27959e71fb2e1f3272dc57abc696f9a3d4e97ce19a"}
Sep 09 20:59:51.179403 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:59:51.179360    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-jnjbr" event={"ID":"6918bd2d-9c6b-48b3-acca-744ea4cae141","Type":"ContainerStarted","Data":"178b1620c1cead62a928c014bc515eb1ecff8a5513c84b5f28b1b4e628164d1e"}
Sep 09 20:59:51.179890 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:59:51.179480    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-64568bcd49-jnjbr"
Sep 09 20:59:51.180973 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:59:51.180952    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-644949856f-cg2jv" event={"ID":"d1ab44fb-91bb-46a4-b592-5a7ef09bc86b","Type":"ContainerStarted","Data":"9b2278c5801f75b0cd9114e5e15ca8c4b11910e5c54c01becf0a06756230df6c"}
Sep 09 20:59:51.181098 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:59:51.181086    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-644949856f-cg2jv"
Sep 09 20:59:51.199010 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:59:51.198966    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-64568bcd49-jnjbr" podStartSLOduration=2.514949741 podStartE2EDuration="6.19895299s" totalImagesPullingTime="3.684003249s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:59:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:59:51.198353731 +0000 UTC m=+448.438447850" watchObservedRunningTime="2026-09-09 20:59:51.19895299 +0000 UTC m=+448.439047108"
Sep 09 20:59:51.219412 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:59:51.219367    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-644949856f-cg2jv" podStartSLOduration=2.573674312 podStartE2EDuration="6.219353411s" totalImagesPullingTime="3.645679099s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:59:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:59:51.217420371 +0000 UTC m=+448.457514501" watchObservedRunningTime="2026-09-09 20:59:51.219353411 +0000 UTC m=+448.459447529"
Sep 09 20:59:57.187288 ip-10-0-138-224 kubenswrapper[2579]: I0909 20:59:57.187252    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-64568bcd49-jnjbr"
Sep 09 21:00:22.189843 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:00:22.189811    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-644949856f-cg2jv"
Sep 09 21:00:30.235084 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:00:30.235037    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-644949856f-cg2jv"]
Sep 09 21:00:30.235759 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:00:30.235266    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve/kserve-controller-manager-644949856f-cg2jv" podUID="d1ab44fb-91bb-46a4-b592-5a7ef09bc86b" containerName="manager" containerID="cri-o://9b2278c5801f75b0cd9114e5e15ca8c4b11910e5c54c01becf0a06756230df6c" gracePeriod=10
Sep 09 21:00:30.272973 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:00:30.272940    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-644949856f-n7kb6"]
Sep 09 21:00:30.277977 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:00:30.277952    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-644949856f-n7kb6"
Sep 09 21:00:30.292090 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:00:30.292066    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-644949856f-n7kb6"]
Sep 09 21:00:30.353758 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:00:30.353679    2579 generic.go:356] "Generic (PLEG): container finished" podID="d1ab44fb-91bb-46a4-b592-5a7ef09bc86b" containerID="9b2278c5801f75b0cd9114e5e15ca8c4b11910e5c54c01becf0a06756230df6c" exitCode=0
Sep 09 21:00:30.353758 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:00:30.353745    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-644949856f-cg2jv" event={"ID":"d1ab44fb-91bb-46a4-b592-5a7ef09bc86b","Type":"ContainerDied","Data":"9b2278c5801f75b0cd9114e5e15ca8c4b11910e5c54c01becf0a06756230df6c"}
Sep 09 21:00:30.385564 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:00:30.385523    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/45a7512f-9c23-4dd5-8507-5ccb327b29a7-cert\") pod \"kserve-controller-manager-644949856f-n7kb6\" (UID: \"45a7512f-9c23-4dd5-8507-5ccb327b29a7\") " pod="kserve/kserve-controller-manager-644949856f-n7kb6"
Sep 09 21:00:30.385734 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:00:30.385624    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzpmf\" (UniqueName: \"kubernetes.io/projected/45a7512f-9c23-4dd5-8507-5ccb327b29a7-kube-api-access-gzpmf\") pod \"kserve-controller-manager-644949856f-n7kb6\" (UID: \"45a7512f-9c23-4dd5-8507-5ccb327b29a7\") " pod="kserve/kserve-controller-manager-644949856f-n7kb6"
Sep 09 21:00:30.477063 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:00:30.477042    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-644949856f-cg2jv"
Sep 09 21:00:30.486581 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:00:30.486541    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/45a7512f-9c23-4dd5-8507-5ccb327b29a7-cert\") pod \"kserve-controller-manager-644949856f-n7kb6\" (UID: \"45a7512f-9c23-4dd5-8507-5ccb327b29a7\") " pod="kserve/kserve-controller-manager-644949856f-n7kb6"
Sep 09 21:00:30.486581 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:00:30.486578    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzpmf\" (UniqueName: \"kubernetes.io/projected/45a7512f-9c23-4dd5-8507-5ccb327b29a7-kube-api-access-gzpmf\") pod \"kserve-controller-manager-644949856f-n7kb6\" (UID: \"45a7512f-9c23-4dd5-8507-5ccb327b29a7\") " pod="kserve/kserve-controller-manager-644949856f-n7kb6"
Sep 09 21:00:30.488733 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:00:30.488712    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/45a7512f-9c23-4dd5-8507-5ccb327b29a7-cert\") pod \"kserve-controller-manager-644949856f-n7kb6\" (UID: \"45a7512f-9c23-4dd5-8507-5ccb327b29a7\") " pod="kserve/kserve-controller-manager-644949856f-n7kb6"
Sep 09 21:00:30.498667 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:00:30.498640    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzpmf\" (UniqueName: \"kubernetes.io/projected/45a7512f-9c23-4dd5-8507-5ccb327b29a7-kube-api-access-gzpmf\") pod \"kserve-controller-manager-644949856f-n7kb6\" (UID: \"45a7512f-9c23-4dd5-8507-5ccb327b29a7\") " pod="kserve/kserve-controller-manager-644949856f-n7kb6"
Sep 09 21:00:30.587766 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:00:30.587731    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dkdqq\" (UniqueName: \"kubernetes.io/projected/d1ab44fb-91bb-46a4-b592-5a7ef09bc86b-kube-api-access-dkdqq\") pod \"d1ab44fb-91bb-46a4-b592-5a7ef09bc86b\" (UID: \"d1ab44fb-91bb-46a4-b592-5a7ef09bc86b\") "
Sep 09 21:00:30.587905 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:00:30.587787    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d1ab44fb-91bb-46a4-b592-5a7ef09bc86b-cert\") pod \"d1ab44fb-91bb-46a4-b592-5a7ef09bc86b\" (UID: \"d1ab44fb-91bb-46a4-b592-5a7ef09bc86b\") "
Sep 09 21:00:30.589972 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:00:30.589940    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1ab44fb-91bb-46a4-b592-5a7ef09bc86b-kube-api-access-dkdqq" (OuterVolumeSpecName: "kube-api-access-dkdqq") pod "d1ab44fb-91bb-46a4-b592-5a7ef09bc86b" (UID: "d1ab44fb-91bb-46a4-b592-5a7ef09bc86b"). InnerVolumeSpecName "kube-api-access-dkdqq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:00:30.589972 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:00:30.589962    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1ab44fb-91bb-46a4-b592-5a7ef09bc86b-cert" (OuterVolumeSpecName: "cert") pod "d1ab44fb-91bb-46a4-b592-5a7ef09bc86b" (UID: "d1ab44fb-91bb-46a4-b592-5a7ef09bc86b"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:00:30.615234 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:00:30.615212    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-644949856f-n7kb6"
Sep 09 21:00:30.688995 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:00:30.688968    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dkdqq\" (UniqueName: \"kubernetes.io/projected/d1ab44fb-91bb-46a4-b592-5a7ef09bc86b-kube-api-access-dkdqq\") on node \"ip-10-0-138-224.ec2.internal\" DevicePath \"\""
Sep 09 21:00:30.688995 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:00:30.688996    2579 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d1ab44fb-91bb-46a4-b592-5a7ef09bc86b-cert\") on node \"ip-10-0-138-224.ec2.internal\" DevicePath \"\""
Sep 09 21:00:30.904130 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:00:30.904107    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-644949856f-n7kb6"]
Sep 09 21:00:30.906857 ip-10-0-138-224 kubenswrapper[2579]: W0909 21:00:30.906832    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45a7512f_9c23_4dd5_8507_5ccb327b29a7.slice/crio-717262378f79e99b87c573cdd67d85be242483bc251b8b489f066ee5fa9f901a WatchSource:0}: Error finding container 717262378f79e99b87c573cdd67d85be242483bc251b8b489f066ee5fa9f901a: Status 404 returned error can't find the container with id 717262378f79e99b87c573cdd67d85be242483bc251b8b489f066ee5fa9f901a
Sep 09 21:00:31.359569 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:00:31.359545    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-644949856f-cg2jv"
Sep 09 21:00:31.359943 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:00:31.359587    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-644949856f-cg2jv" event={"ID":"d1ab44fb-91bb-46a4-b592-5a7ef09bc86b","Type":"ContainerDied","Data":"d116632a76ba2527cb26466d75c971e483445dab589c04dc5c1be21b91b42758"}
Sep 09 21:00:31.359943 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:00:31.359648    2579 scope.go:117] "RemoveContainer" containerID="9b2278c5801f75b0cd9114e5e15ca8c4b11910e5c54c01becf0a06756230df6c"
Sep 09 21:00:31.361025 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:00:31.360877    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-644949856f-n7kb6" event={"ID":"45a7512f-9c23-4dd5-8507-5ccb327b29a7","Type":"ContainerStarted","Data":"717262378f79e99b87c573cdd67d85be242483bc251b8b489f066ee5fa9f901a"}
Sep 09 21:00:31.384878 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:00:31.384858    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-644949856f-cg2jv"]
Sep 09 21:00:31.390942 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:00:31.390924    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/kserve-controller-manager-644949856f-cg2jv"]
Sep 09 21:00:32.368129 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:00:32.368096    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-644949856f-n7kb6" event={"ID":"45a7512f-9c23-4dd5-8507-5ccb327b29a7","Type":"ContainerStarted","Data":"eae3b9105afe7070adee623e9f80ef0adc5b0614eff7ea9602dcb2b57ff091d4"}
Sep 09 21:00:32.368606 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:00:32.368205    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-644949856f-n7kb6"
Sep 09 21:00:32.401030 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:00:32.400977    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-644949856f-n7kb6" podStartSLOduration=1.925188304 podStartE2EDuration="2.400962456s" totalImagesPullingTime="475.774152ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:00:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:00:32.400000485 +0000 UTC m=+489.640094602" watchObservedRunningTime="2026-09-09 21:00:32.400962456 +0000 UTC m=+489.641056571"
Sep 09 21:00:33.290721 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:00:33.290688    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d1ab44fb-91bb-46a4-b592-5a7ef09bc86b" path="/var/lib/kubelet/pods/d1ab44fb-91bb-46a4-b592-5a7ef09bc86b/volumes"
Sep 09 21:01:03.378002 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:01:03.377972    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-644949856f-n7kb6"
Sep 09 21:01:04.738416 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:01:04.738386    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-7b966d7984-znm4k"]
Sep 09 21:01:04.738809 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:01:04.738728    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d1ab44fb-91bb-46a4-b592-5a7ef09bc86b" containerName="manager"
Sep 09 21:01:04.738809 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:01:04.738741    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="d1ab44fb-91bb-46a4-b592-5a7ef09bc86b" containerName="manager"
Sep 09 21:01:04.738809 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:01:04.738790    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="d1ab44fb-91bb-46a4-b592-5a7ef09bc86b" containerName="manager"
Sep 09 21:01:04.741810 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:01:04.741793    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-7b966d7984-znm4k"
Sep 09 21:01:04.745838 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:01:04.745819    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\""
Sep 09 21:01:04.745903 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:01:04.745818    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-metrics-tls\""
Sep 09 21:01:04.745948 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:01:04.745821    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-9fml9\""
Sep 09 21:01:04.754192 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:01:04.754169    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-7b966d7984-znm4k"]
Sep 09 21:01:04.837042 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:01:04.837011    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/859b8655-db7f-43c6-a775-7d06fc7a212b-cert\") pod \"odh-model-controller-7b966d7984-znm4k\" (UID: \"859b8655-db7f-43c6-a775-7d06fc7a212b\") " pod="kserve/odh-model-controller-7b966d7984-znm4k"
Sep 09 21:01:04.837235 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:01:04.837060    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bx7vl\" (UniqueName: \"kubernetes.io/projected/859b8655-db7f-43c6-a775-7d06fc7a212b-kube-api-access-bx7vl\") pod \"odh-model-controller-7b966d7984-znm4k\" (UID: \"859b8655-db7f-43c6-a775-7d06fc7a212b\") " pod="kserve/odh-model-controller-7b966d7984-znm4k"
Sep 09 21:01:04.837235 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:01:04.837104    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/859b8655-db7f-43c6-a775-7d06fc7a212b-metrics-certs\") pod \"odh-model-controller-7b966d7984-znm4k\" (UID: \"859b8655-db7f-43c6-a775-7d06fc7a212b\") " pod="kserve/odh-model-controller-7b966d7984-znm4k"
Sep 09 21:01:04.937907 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:01:04.937866    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/859b8655-db7f-43c6-a775-7d06fc7a212b-cert\") pod \"odh-model-controller-7b966d7984-znm4k\" (UID: \"859b8655-db7f-43c6-a775-7d06fc7a212b\") " pod="kserve/odh-model-controller-7b966d7984-znm4k"
Sep 09 21:01:04.938076 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:01:04.937920    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bx7vl\" (UniqueName: \"kubernetes.io/projected/859b8655-db7f-43c6-a775-7d06fc7a212b-kube-api-access-bx7vl\") pod \"odh-model-controller-7b966d7984-znm4k\" (UID: \"859b8655-db7f-43c6-a775-7d06fc7a212b\") " pod="kserve/odh-model-controller-7b966d7984-znm4k"
Sep 09 21:01:04.938076 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:01:04.937949    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/859b8655-db7f-43c6-a775-7d06fc7a212b-metrics-certs\") pod \"odh-model-controller-7b966d7984-znm4k\" (UID: \"859b8655-db7f-43c6-a775-7d06fc7a212b\") " pod="kserve/odh-model-controller-7b966d7984-znm4k"
Sep 09 21:01:04.940683 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:01:04.940660    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/859b8655-db7f-43c6-a775-7d06fc7a212b-metrics-certs\") pod \"odh-model-controller-7b966d7984-znm4k\" (UID: \"859b8655-db7f-43c6-a775-7d06fc7a212b\") " pod="kserve/odh-model-controller-7b966d7984-znm4k"
Sep 09 21:01:04.940779 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:01:04.940704    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/859b8655-db7f-43c6-a775-7d06fc7a212b-cert\") pod \"odh-model-controller-7b966d7984-znm4k\" (UID: \"859b8655-db7f-43c6-a775-7d06fc7a212b\") " pod="kserve/odh-model-controller-7b966d7984-znm4k"
Sep 09 21:01:04.949879 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:01:04.949849    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bx7vl\" (UniqueName: \"kubernetes.io/projected/859b8655-db7f-43c6-a775-7d06fc7a212b-kube-api-access-bx7vl\") pod \"odh-model-controller-7b966d7984-znm4k\" (UID: \"859b8655-db7f-43c6-a775-7d06fc7a212b\") " pod="kserve/odh-model-controller-7b966d7984-znm4k"
Sep 09 21:01:05.052327 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:01:05.052242    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-7b966d7984-znm4k"
Sep 09 21:01:05.325397 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:01:05.325375    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-7b966d7984-znm4k"]
Sep 09 21:01:05.327870 ip-10-0-138-224 kubenswrapper[2579]: W0909 21:01:05.327840    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod859b8655_db7f_43c6_a775_7d06fc7a212b.slice/crio-8361f856a1a0e2e2b5159862caf078595ad747ff672fc329e2d0736aa721fcf7 WatchSource:0}: Error finding container 8361f856a1a0e2e2b5159862caf078595ad747ff672fc329e2d0736aa721fcf7: Status 404 returned error can't find the container with id 8361f856a1a0e2e2b5159862caf078595ad747ff672fc329e2d0736aa721fcf7
Sep 09 21:01:05.508409 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:01:05.508366    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-7b966d7984-znm4k" event={"ID":"859b8655-db7f-43c6-a775-7d06fc7a212b","Type":"ContainerStarted","Data":"8361f856a1a0e2e2b5159862caf078595ad747ff672fc329e2d0736aa721fcf7"}
Sep 09 21:01:09.529364 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:01:09.529328    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-7b966d7984-znm4k" event={"ID":"859b8655-db7f-43c6-a775-7d06fc7a212b","Type":"ContainerStarted","Data":"93c66c874f3f59dd33233f335e723fdab87375336006264ab06b1a9c5a607dc7"}
Sep 09 21:01:09.529807 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:01:09.529428    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-7b966d7984-znm4k"
Sep 09 21:01:09.552045 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:01:09.549313    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/odh-model-controller-7b966d7984-znm4k" podStartSLOduration=2.379898617 podStartE2EDuration="5.54929336s" totalImagesPullingTime="3.169394743s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:01:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:01:09.546175057 +0000 UTC m=+526.786269176" watchObservedRunningTime="2026-09-09 21:01:09.54929336 +0000 UTC m=+526.789387479"
Sep 09 21:01:20.535152 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:01:20.535082    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-7b966d7984-znm4k"
Sep 09 21:02:52.808475 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:02:52.808393    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-8df9t"]
Sep 09 21:02:52.811885 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:02:52.811863    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-8df9t"
Sep 09 21:02:52.814708 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:02:52.814685    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 09 21:02:52.814933 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:02:52.814916    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 09 21:02:52.815032 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:02:52.815016    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-kube-rbac-proxy-sar-config\""
Sep 09 21:02:52.815100 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:02:52.815026    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 09 21:02:52.815158 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:02:52.815122    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-predictor-serving-cert\""
Sep 09 21:02:52.816174 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:02:52.816156    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 09 21:02:52.816282 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:02:52.816232    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-tlmnz\""
Sep 09 21:02:52.820940 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:02:52.820375    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-8df9t"]
Sep 09 21:02:52.901503 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:02:52.901472    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/782c0960-aa4e-47df-a18b-5e251e9e3106-cabundle-cert\") pod \"isvc-canary-rollback-predictor-6cb68d45bc-8df9t\" (UID: \"782c0960-aa4e-47df-a18b-5e251e9e3106\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-8df9t"
Sep 09 21:02:52.901667 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:02:52.901537    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vtsmf\" (UniqueName: \"kubernetes.io/projected/782c0960-aa4e-47df-a18b-5e251e9e3106-kube-api-access-vtsmf\") pod \"isvc-canary-rollback-predictor-6cb68d45bc-8df9t\" (UID: \"782c0960-aa4e-47df-a18b-5e251e9e3106\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-8df9t"
Sep 09 21:02:52.901667 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:02:52.901562    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/782c0960-aa4e-47df-a18b-5e251e9e3106-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-6cb68d45bc-8df9t\" (UID: \"782c0960-aa4e-47df-a18b-5e251e9e3106\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-8df9t"
Sep 09 21:02:52.901667 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:02:52.901584    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/782c0960-aa4e-47df-a18b-5e251e9e3106-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-6cb68d45bc-8df9t\" (UID: \"782c0960-aa4e-47df-a18b-5e251e9e3106\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-8df9t"
Sep 09 21:02:52.901667 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:02:52.901608    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/782c0960-aa4e-47df-a18b-5e251e9e3106-proxy-tls\") pod \"isvc-canary-rollback-predictor-6cb68d45bc-8df9t\" (UID: \"782c0960-aa4e-47df-a18b-5e251e9e3106\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-8df9t"
Sep 09 21:02:52.911385 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:02:52.911358    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2a6f9-predictor-86589975f4-jprmq"]
Sep 09 21:02:52.914804 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:02:52.914786    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2a6f9-predictor-86589975f4-jprmq"
Sep 09 21:02:52.917511 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:02:52.917492    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-raw-sklearn-batcher-2a6f9-predictor-serving-cert\""
Sep 09 21:02:52.917597 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:02:52.917543    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-raw-sklearn-batcher-2a6f9-kube-rbac-proxy-sar-config\""
Sep 09 21:02:52.924296 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:02:52.924277    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2a6f9-predictor-86589975f4-jprmq"]
Sep 09 21:02:53.002908 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:02:53.002874    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/782c0960-aa4e-47df-a18b-5e251e9e3106-proxy-tls\") pod \"isvc-canary-rollback-predictor-6cb68d45bc-8df9t\" (UID: \"782c0960-aa4e-47df-a18b-5e251e9e3106\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-8df9t"
Sep 09 21:02:53.003049 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:02:53.002950    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c9e563c3-c2c7-44c1-8061-f7730429b4ee-proxy-tls\") pod \"isvc-raw-sklearn-batcher-2a6f9-predictor-86589975f4-jprmq\" (UID: \"c9e563c3-c2c7-44c1-8061-f7730429b4ee\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2a6f9-predictor-86589975f4-jprmq"
Sep 09 21:02:53.003118 ip-10-0-138-224 kubenswrapper[2579]: E0909 21:02:53.003043    2579 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-rollback-predictor-serving-cert: secret "isvc-canary-rollback-predictor-serving-cert" not found
Sep 09 21:02:53.003118 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:02:53.003072    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/782c0960-aa4e-47df-a18b-5e251e9e3106-cabundle-cert\") pod \"isvc-canary-rollback-predictor-6cb68d45bc-8df9t\" (UID: \"782c0960-aa4e-47df-a18b-5e251e9e3106\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-8df9t"
Sep 09 21:02:53.003118 ip-10-0-138-224 kubenswrapper[2579]: E0909 21:02:53.003116    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/782c0960-aa4e-47df-a18b-5e251e9e3106-proxy-tls podName:782c0960-aa4e-47df-a18b-5e251e9e3106 nodeName:}" failed. No retries permitted until 2026-09-09 21:02:53.503094965 +0000 UTC m=+630.743189067 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/782c0960-aa4e-47df-a18b-5e251e9e3106-proxy-tls") pod "isvc-canary-rollback-predictor-6cb68d45bc-8df9t" (UID: "782c0960-aa4e-47df-a18b-5e251e9e3106") : secret "isvc-canary-rollback-predictor-serving-cert" not found
Sep 09 21:02:53.003279 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:02:53.003138    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c9e563c3-c2c7-44c1-8061-f7730429b4ee-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-2a6f9-predictor-86589975f4-jprmq\" (UID: \"c9e563c3-c2c7-44c1-8061-f7730429b4ee\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2a6f9-predictor-86589975f4-jprmq"
Sep 09 21:02:53.003279 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:02:53.003173    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c9e563c3-c2c7-44c1-8061-f7730429b4ee-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-2a6f9-predictor-86589975f4-jprmq\" (UID: \"c9e563c3-c2c7-44c1-8061-f7730429b4ee\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2a6f9-predictor-86589975f4-jprmq"
Sep 09 21:02:53.003279 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:02:53.003206    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtsmf\" (UniqueName: \"kubernetes.io/projected/782c0960-aa4e-47df-a18b-5e251e9e3106-kube-api-access-vtsmf\") pod \"isvc-canary-rollback-predictor-6cb68d45bc-8df9t\" (UID: \"782c0960-aa4e-47df-a18b-5e251e9e3106\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-8df9t"
Sep 09 21:02:53.003279 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:02:53.003233    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c9e563c3-c2c7-44c1-8061-f7730429b4ee-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-2a6f9-predictor-86589975f4-jprmq\" (UID: \"c9e563c3-c2c7-44c1-8061-f7730429b4ee\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2a6f9-predictor-86589975f4-jprmq"
Sep 09 21:02:53.003279 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:02:53.003261    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sf979\" (UniqueName: \"kubernetes.io/projected/c9e563c3-c2c7-44c1-8061-f7730429b4ee-kube-api-access-sf979\") pod \"isvc-raw-sklearn-batcher-2a6f9-predictor-86589975f4-jprmq\" (UID: \"c9e563c3-c2c7-44c1-8061-f7730429b4ee\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2a6f9-predictor-86589975f4-jprmq"
Sep 09 21:02:53.003576 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:02:53.003292    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/782c0960-aa4e-47df-a18b-5e251e9e3106-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-6cb68d45bc-8df9t\" (UID: \"782c0960-aa4e-47df-a18b-5e251e9e3106\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-8df9t"
Sep 09 21:02:53.003576 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:02:53.003321    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/782c0960-aa4e-47df-a18b-5e251e9e3106-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-6cb68d45bc-8df9t\" (UID: \"782c0960-aa4e-47df-a18b-5e251e9e3106\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-8df9t"
Sep 09 21:02:53.003693 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:02:53.003674    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/782c0960-aa4e-47df-a18b-5e251e9e3106-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-6cb68d45bc-8df9t\" (UID: \"782c0960-aa4e-47df-a18b-5e251e9e3106\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-8df9t"
Sep 09 21:02:53.003842 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:02:53.003820    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/782c0960-aa4e-47df-a18b-5e251e9e3106-cabundle-cert\") pod \"isvc-canary-rollback-predictor-6cb68d45bc-8df9t\" (UID: \"782c0960-aa4e-47df-a18b-5e251e9e3106\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-8df9t"
Sep 09 21:02:53.004214 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:02:53.004192    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/782c0960-aa4e-47df-a18b-5e251e9e3106-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-6cb68d45bc-8df9t\" (UID: \"782c0960-aa4e-47df-a18b-5e251e9e3106\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-8df9t"
Sep 09 21:02:53.013321 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:02:53.013297    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtsmf\" (UniqueName: \"kubernetes.io/projected/782c0960-aa4e-47df-a18b-5e251e9e3106-kube-api-access-vtsmf\") pod \"isvc-canary-rollback-predictor-6cb68d45bc-8df9t\" (UID: \"782c0960-aa4e-47df-a18b-5e251e9e3106\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-8df9t"
Sep 09 21:02:53.104621 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:02:53.104541    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c9e563c3-c2c7-44c1-8061-f7730429b4ee-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-2a6f9-predictor-86589975f4-jprmq\" (UID: \"c9e563c3-c2c7-44c1-8061-f7730429b4ee\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2a6f9-predictor-86589975f4-jprmq"
Sep 09 21:02:53.104621 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:02:53.104577    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c9e563c3-c2c7-44c1-8061-f7730429b4ee-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-2a6f9-predictor-86589975f4-jprmq\" (UID: \"c9e563c3-c2c7-44c1-8061-f7730429b4ee\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2a6f9-predictor-86589975f4-jprmq"
Sep 09 21:02:53.104621 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:02:53.104600    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c9e563c3-c2c7-44c1-8061-f7730429b4ee-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-2a6f9-predictor-86589975f4-jprmq\" (UID: \"c9e563c3-c2c7-44c1-8061-f7730429b4ee\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2a6f9-predictor-86589975f4-jprmq"
Sep 09 21:02:53.104621 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:02:53.104616    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sf979\" (UniqueName: \"kubernetes.io/projected/c9e563c3-c2c7-44c1-8061-f7730429b4ee-kube-api-access-sf979\") pod \"isvc-raw-sklearn-batcher-2a6f9-predictor-86589975f4-jprmq\" (UID: \"c9e563c3-c2c7-44c1-8061-f7730429b4ee\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2a6f9-predictor-86589975f4-jprmq"
Sep 09 21:02:53.105146 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:02:53.104809    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c9e563c3-c2c7-44c1-8061-f7730429b4ee-proxy-tls\") pod \"isvc-raw-sklearn-batcher-2a6f9-predictor-86589975f4-jprmq\" (UID: \"c9e563c3-c2c7-44c1-8061-f7730429b4ee\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2a6f9-predictor-86589975f4-jprmq"
Sep 09 21:02:53.105146 ip-10-0-138-224 kubenswrapper[2579]: E0909 21:02:53.104965    2579 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2a6f9-predictor-serving-cert: secret "isvc-raw-sklearn-batcher-2a6f9-predictor-serving-cert" not found
Sep 09 21:02:53.105146 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:02:53.105020    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c9e563c3-c2c7-44c1-8061-f7730429b4ee-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-2a6f9-predictor-86589975f4-jprmq\" (UID: \"c9e563c3-c2c7-44c1-8061-f7730429b4ee\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2a6f9-predictor-86589975f4-jprmq"
Sep 09 21:02:53.105146 ip-10-0-138-224 kubenswrapper[2579]: E0909 21:02:53.105044    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c9e563c3-c2c7-44c1-8061-f7730429b4ee-proxy-tls podName:c9e563c3-c2c7-44c1-8061-f7730429b4ee nodeName:}" failed. No retries permitted until 2026-09-09 21:02:53.605025724 +0000 UTC m=+630.845119838 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/c9e563c3-c2c7-44c1-8061-f7730429b4ee-proxy-tls") pod "isvc-raw-sklearn-batcher-2a6f9-predictor-86589975f4-jprmq" (UID: "c9e563c3-c2c7-44c1-8061-f7730429b4ee") : secret "isvc-raw-sklearn-batcher-2a6f9-predictor-serving-cert" not found
Sep 09 21:02:53.105368 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:02:53.105347    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c9e563c3-c2c7-44c1-8061-f7730429b4ee-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-2a6f9-predictor-86589975f4-jprmq\" (UID: \"c9e563c3-c2c7-44c1-8061-f7730429b4ee\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2a6f9-predictor-86589975f4-jprmq"
Sep 09 21:02:53.105410 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:02:53.105358    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c9e563c3-c2c7-44c1-8061-f7730429b4ee-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-2a6f9-predictor-86589975f4-jprmq\" (UID: \"c9e563c3-c2c7-44c1-8061-f7730429b4ee\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2a6f9-predictor-86589975f4-jprmq"
Sep 09 21:02:53.124612 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:02:53.124578    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sf979\" (UniqueName: \"kubernetes.io/projected/c9e563c3-c2c7-44c1-8061-f7730429b4ee-kube-api-access-sf979\") pod \"isvc-raw-sklearn-batcher-2a6f9-predictor-86589975f4-jprmq\" (UID: \"c9e563c3-c2c7-44c1-8061-f7730429b4ee\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2a6f9-predictor-86589975f4-jprmq"
Sep 09 21:02:53.508325 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:02:53.508288    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/782c0960-aa4e-47df-a18b-5e251e9e3106-proxy-tls\") pod \"isvc-canary-rollback-predictor-6cb68d45bc-8df9t\" (UID: \"782c0960-aa4e-47df-a18b-5e251e9e3106\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-8df9t"
Sep 09 21:02:53.510653 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:02:53.510624    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/782c0960-aa4e-47df-a18b-5e251e9e3106-proxy-tls\") pod \"isvc-canary-rollback-predictor-6cb68d45bc-8df9t\" (UID: \"782c0960-aa4e-47df-a18b-5e251e9e3106\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-8df9t"
Sep 09 21:02:53.609532 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:02:53.609495    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c9e563c3-c2c7-44c1-8061-f7730429b4ee-proxy-tls\") pod \"isvc-raw-sklearn-batcher-2a6f9-predictor-86589975f4-jprmq\" (UID: \"c9e563c3-c2c7-44c1-8061-f7730429b4ee\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2a6f9-predictor-86589975f4-jprmq"
Sep 09 21:02:53.611864 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:02:53.611839    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c9e563c3-c2c7-44c1-8061-f7730429b4ee-proxy-tls\") pod \"isvc-raw-sklearn-batcher-2a6f9-predictor-86589975f4-jprmq\" (UID: \"c9e563c3-c2c7-44c1-8061-f7730429b4ee\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2a6f9-predictor-86589975f4-jprmq"
Sep 09 21:02:53.725601 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:02:53.725571    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-8df9t"
Sep 09 21:02:53.826110 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:02:53.826077    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2a6f9-predictor-86589975f4-jprmq"
Sep 09 21:02:53.999087 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:02:53.999064    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-8df9t"]
Sep 09 21:02:54.001475 ip-10-0-138-224 kubenswrapper[2579]: W0909 21:02:54.001421    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod782c0960_aa4e_47df_a18b_5e251e9e3106.slice/crio-a574b99faa67634c84ab4c97720b31361c4c0f437d28fd7ef6755b8c521ef4c2 WatchSource:0}: Error finding container a574b99faa67634c84ab4c97720b31361c4c0f437d28fd7ef6755b8c521ef4c2: Status 404 returned error can't find the container with id a574b99faa67634c84ab4c97720b31361c4c0f437d28fd7ef6755b8c521ef4c2
Sep 09 21:02:54.098805 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:02:54.098781    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2a6f9-predictor-86589975f4-jprmq"]
Sep 09 21:02:54.101629 ip-10-0-138-224 kubenswrapper[2579]: W0909 21:02:54.101584    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9e563c3_c2c7_44c1_8061_f7730429b4ee.slice/crio-c1f4f95834a53a2272452e3c362c9f2e57d2ad9a46f4f3219c4da6c17fb525d7 WatchSource:0}: Error finding container c1f4f95834a53a2272452e3c362c9f2e57d2ad9a46f4f3219c4da6c17fb525d7: Status 404 returned error can't find the container with id c1f4f95834a53a2272452e3c362c9f2e57d2ad9a46f4f3219c4da6c17fb525d7
Sep 09 21:02:54.997323 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:02:54.997288    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-8df9t" event={"ID":"782c0960-aa4e-47df-a18b-5e251e9e3106","Type":"ContainerStarted","Data":"a574b99faa67634c84ab4c97720b31361c4c0f437d28fd7ef6755b8c521ef4c2"}
Sep 09 21:02:54.998607 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:02:54.998585    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2a6f9-predictor-86589975f4-jprmq" event={"ID":"c9e563c3-c2c7-44c1-8061-f7730429b4ee","Type":"ContainerStarted","Data":"c1f4f95834a53a2272452e3c362c9f2e57d2ad9a46f4f3219c4da6c17fb525d7"}
Sep 09 21:02:58.017110 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:02:58.017074    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2a6f9-predictor-86589975f4-jprmq" event={"ID":"c9e563c3-c2c7-44c1-8061-f7730429b4ee","Type":"ContainerStarted","Data":"3fae88b1cd6d0d8508909885184965fef5a4f35d1aed08c7dd1a4ec45c29e6b0"}
Sep 09 21:02:58.018976 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:02:58.018947    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-8df9t" event={"ID":"782c0960-aa4e-47df-a18b-5e251e9e3106","Type":"ContainerStarted","Data":"fb057de33235899677d846f2fd717e84f2f9b9d2112d96de1c0dbeaf674818f0"}
Sep 09 21:03:02.038621 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:03:02.038590    2579 generic.go:356] "Generic (PLEG): container finished" podID="782c0960-aa4e-47df-a18b-5e251e9e3106" containerID="fb057de33235899677d846f2fd717e84f2f9b9d2112d96de1c0dbeaf674818f0" exitCode=0
Sep 09 21:03:02.039077 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:03:02.038666    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-8df9t" event={"ID":"782c0960-aa4e-47df-a18b-5e251e9e3106","Type":"ContainerDied","Data":"fb057de33235899677d846f2fd717e84f2f9b9d2112d96de1c0dbeaf674818f0"}
Sep 09 21:03:02.040346 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:03:02.040328    2579 generic.go:356] "Generic (PLEG): container finished" podID="c9e563c3-c2c7-44c1-8061-f7730429b4ee" containerID="3fae88b1cd6d0d8508909885184965fef5a4f35d1aed08c7dd1a4ec45c29e6b0" exitCode=0
Sep 09 21:03:02.040407 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:03:02.040365    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2a6f9-predictor-86589975f4-jprmq" event={"ID":"c9e563c3-c2c7-44c1-8061-f7730429b4ee","Type":"ContainerDied","Data":"3fae88b1cd6d0d8508909885184965fef5a4f35d1aed08c7dd1a4ec45c29e6b0"}
Sep 09 21:03:18.583249 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:03:18.583230    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 21:03:19.174367 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:03:19.174242    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2a6f9-predictor-86589975f4-jprmq" event={"ID":"c9e563c3-c2c7-44c1-8061-f7730429b4ee","Type":"ContainerStarted","Data":"5bf007785187b3f23ae9ffb8f69b669c4f8b9b04a6816ab3faa31ad35ef7c8d7"}
Sep 09 21:03:19.180408 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:03:19.180330    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-8df9t" event={"ID":"782c0960-aa4e-47df-a18b-5e251e9e3106","Type":"ContainerStarted","Data":"3d261d27bca6217b7885a21f98d958fdce3942fa81b7e7a8d791a7bec8ae3ffb"}
Sep 09 21:03:22.198984 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:03:22.198946    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-8df9t" event={"ID":"782c0960-aa4e-47df-a18b-5e251e9e3106","Type":"ContainerStarted","Data":"fd803a1f8c7ba9421e20b610399b4af991d0ce08bd6b0953cf98761ddabe1658"}
Sep 09 21:03:22.199475 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:03:22.199028    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-8df9t"
Sep 09 21:03:22.199475 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:03:22.199197    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-8df9t"
Sep 09 21:03:22.200821 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:03:22.200784    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-8df9t" podUID="782c0960-aa4e-47df-a18b-5e251e9e3106" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Sep 09 21:03:22.201545 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:03:22.201524    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2a6f9-predictor-86589975f4-jprmq" event={"ID":"c9e563c3-c2c7-44c1-8061-f7730429b4ee","Type":"ContainerStarted","Data":"45c3765997542ba42dd0dcbf0fddf8a6c1e09718453cc85f2d2b907852304d27"}
Sep 09 21:03:22.221245 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:03:22.221205    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-8df9t" podStartSLOduration=3.297707804 podStartE2EDuration="30.221191883s" totalImagesPullingTime="23.072218992s" totalInitContainerRuntime="3.851265087s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:02:52 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:03:22.22011862 +0000 UTC m=+659.460212739" watchObservedRunningTime="2026-09-09 21:03:22.221191883 +0000 UTC m=+659.461286064"
Sep 09 21:03:23.207360 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:03:23.207324    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-8df9t" podUID="782c0960-aa4e-47df-a18b-5e251e9e3106" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Sep 09 21:03:25.219655 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:03:25.219615    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2a6f9-predictor-86589975f4-jprmq" event={"ID":"c9e563c3-c2c7-44c1-8061-f7730429b4ee","Type":"ContainerStarted","Data":"9f52b3f4b8b5a088dae748eb8d7f33245ba2a46f936572e7a33aaeca5881579e"}
Sep 09 21:03:25.220084 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:03:25.219728    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2a6f9-predictor-86589975f4-jprmq"
Sep 09 21:03:25.220084 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:03:25.219766    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2a6f9-predictor-86589975f4-jprmq"
Sep 09 21:03:25.221180 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:03:25.221150    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2a6f9-predictor-86589975f4-jprmq" podUID="c9e563c3-c2c7-44c1-8061-f7730429b4ee" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 09 21:03:25.241698 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:03:25.241656    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2a6f9-predictor-86589975f4-jprmq" podStartSLOduration=3.31757059 podStartE2EDuration="33.241642337s" totalImagesPullingTime="26.19169854s" totalInitContainerRuntime="3.732373207s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:02:52 +0000 UTC" imagePullSessionsCount=4 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:03:25.241265652 +0000 UTC m=+662.481359796" watchObservedRunningTime="2026-09-09 21:03:25.241642337 +0000 UTC m=+662.481736455"
Sep 09 21:03:26.223861 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:03:26.223814    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2a6f9-predictor-86589975f4-jprmq"
Sep 09 21:03:26.224305 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:03:26.223998    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2a6f9-predictor-86589975f4-jprmq" podUID="c9e563c3-c2c7-44c1-8061-f7730429b4ee" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 09 21:03:26.224923 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:03:26.224897    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2a6f9-predictor-86589975f4-jprmq" podUID="c9e563c3-c2c7-44c1-8061-f7730429b4ee" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 21:03:26.228137 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:03:26.228117    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2a6f9-predictor-86589975f4-jprmq"
Sep 09 21:03:27.229821 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:03:27.229776    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2a6f9-predictor-86589975f4-jprmq" podUID="c9e563c3-c2c7-44c1-8061-f7730429b4ee" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 09 21:03:27.230233 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:03:27.230059    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2a6f9-predictor-86589975f4-jprmq" podUID="c9e563c3-c2c7-44c1-8061-f7730429b4ee" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 21:03:28.212626 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:03:28.212597    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-8df9t"
Sep 09 21:03:28.213146 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:03:28.213122    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-8df9t" podUID="782c0960-aa4e-47df-a18b-5e251e9e3106" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Sep 09 21:03:28.235015 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:03:28.234981    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2a6f9-predictor-86589975f4-jprmq" podUID="c9e563c3-c2c7-44c1-8061-f7730429b4ee" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 09 21:03:28.235431 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:03:28.235336    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2a6f9-predictor-86589975f4-jprmq" podUID="c9e563c3-c2c7-44c1-8061-f7730429b4ee" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 21:03:38.213730 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:03:38.213685    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-8df9t" podUID="782c0960-aa4e-47df-a18b-5e251e9e3106" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Sep 09 21:03:38.235676 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:03:38.235645    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2a6f9-predictor-86589975f4-jprmq" podUID="c9e563c3-c2c7-44c1-8061-f7730429b4ee" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 09 21:03:38.236215 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:03:38.236188    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2a6f9-predictor-86589975f4-jprmq" podUID="c9e563c3-c2c7-44c1-8061-f7730429b4ee" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 21:03:48.213629 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:03:48.213589    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-8df9t" podUID="782c0960-aa4e-47df-a18b-5e251e9e3106" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Sep 09 21:03:48.235661 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:03:48.235625    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2a6f9-predictor-86589975f4-jprmq" podUID="c9e563c3-c2c7-44c1-8061-f7730429b4ee" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 09 21:03:48.236172 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:03:48.236141    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2a6f9-predictor-86589975f4-jprmq" podUID="c9e563c3-c2c7-44c1-8061-f7730429b4ee" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 21:03:58.213329 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:03:58.213285    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-8df9t" podUID="782c0960-aa4e-47df-a18b-5e251e9e3106" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Sep 09 21:03:58.234927 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:03:58.234897    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2a6f9-predictor-86589975f4-jprmq" podUID="c9e563c3-c2c7-44c1-8061-f7730429b4ee" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 09 21:03:58.235386 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:03:58.235356    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2a6f9-predictor-86589975f4-jprmq" podUID="c9e563c3-c2c7-44c1-8061-f7730429b4ee" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 21:04:08.213747 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:04:08.213703    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-8df9t" podUID="782c0960-aa4e-47df-a18b-5e251e9e3106" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Sep 09 21:04:08.235002 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:04:08.234964    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2a6f9-predictor-86589975f4-jprmq" podUID="c9e563c3-c2c7-44c1-8061-f7730429b4ee" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 09 21:04:08.235415 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:04:08.235395    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2a6f9-predictor-86589975f4-jprmq" podUID="c9e563c3-c2c7-44c1-8061-f7730429b4ee" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 21:04:18.213198 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:04:18.213104    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-8df9t" podUID="782c0960-aa4e-47df-a18b-5e251e9e3106" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Sep 09 21:04:18.235119 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:04:18.235093    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2a6f9-predictor-86589975f4-jprmq" podUID="c9e563c3-c2c7-44c1-8061-f7730429b4ee" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 09 21:04:18.235546 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:04:18.235523    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2a6f9-predictor-86589975f4-jprmq" podUID="c9e563c3-c2c7-44c1-8061-f7730429b4ee" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 21:04:28.213979 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:04:28.213939    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-8df9t" podUID="782c0960-aa4e-47df-a18b-5e251e9e3106" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Sep 09 21:04:28.235220 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:04:28.235190    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2a6f9-predictor-86589975f4-jprmq" podUID="c9e563c3-c2c7-44c1-8061-f7730429b4ee" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 09 21:04:28.235667 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:04:28.235644    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2a6f9-predictor-86589975f4-jprmq" podUID="c9e563c3-c2c7-44c1-8061-f7730429b4ee" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 21:04:35.051834 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:04:35.051799    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-8df9t"]
Sep 09 21:04:35.052297 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:04:35.052032    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-8df9t" podUID="782c0960-aa4e-47df-a18b-5e251e9e3106" containerName="kserve-container" containerID="cri-o://3d261d27bca6217b7885a21f98d958fdce3942fa81b7e7a8d791a7bec8ae3ffb" gracePeriod=30
Sep 09 21:04:35.052297 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:04:35.052131    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-8df9t" podUID="782c0960-aa4e-47df-a18b-5e251e9e3106" containerName="kube-rbac-proxy" containerID="cri-o://fd803a1f8c7ba9421e20b610399b4af991d0ce08bd6b0953cf98761ddabe1658" gracePeriod=30
Sep 09 21:04:35.052817 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:04:35.052637    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-8df9t"
Sep 09 21:04:35.533392 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:04:35.533357    2579 generic.go:356] "Generic (PLEG): container finished" podID="782c0960-aa4e-47df-a18b-5e251e9e3106" containerID="fd803a1f8c7ba9421e20b610399b4af991d0ce08bd6b0953cf98761ddabe1658" exitCode=2
Sep 09 21:04:35.533582 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:04:35.533464    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-8df9t" event={"ID":"782c0960-aa4e-47df-a18b-5e251e9e3106","Type":"ContainerDied","Data":"fd803a1f8c7ba9421e20b610399b4af991d0ce08bd6b0953cf98761ddabe1658"}
Sep 09 21:04:38.208600 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:04:38.208557    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-8df9t" podUID="782c0960-aa4e-47df-a18b-5e251e9e3106" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.30:8643/healthz\": dial tcp 10.134.0.30:8643: connect: connection refused"
Sep 09 21:04:38.235701 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:04:38.235671    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2a6f9-predictor-86589975f4-jprmq"
Sep 09 21:04:38.235949 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:04:38.235933    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2a6f9-predictor-86589975f4-jprmq"
Sep 09 21:04:39.895234 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:04:39.895200    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-8df9t"
Sep 09 21:04:40.027779 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:04:40.027671    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vtsmf\" (UniqueName: \"kubernetes.io/projected/782c0960-aa4e-47df-a18b-5e251e9e3106-kube-api-access-vtsmf\") pod \"782c0960-aa4e-47df-a18b-5e251e9e3106\" (UID: \"782c0960-aa4e-47df-a18b-5e251e9e3106\") "
Sep 09 21:04:40.027779 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:04:40.027725    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/782c0960-aa4e-47df-a18b-5e251e9e3106-cabundle-cert\") pod \"782c0960-aa4e-47df-a18b-5e251e9e3106\" (UID: \"782c0960-aa4e-47df-a18b-5e251e9e3106\") "
Sep 09 21:04:40.028009 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:04:40.027802    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/782c0960-aa4e-47df-a18b-5e251e9e3106-kube-rbac-proxy-sar-config\") pod \"782c0960-aa4e-47df-a18b-5e251e9e3106\" (UID: \"782c0960-aa4e-47df-a18b-5e251e9e3106\") "
Sep 09 21:04:40.028009 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:04:40.027827    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/782c0960-aa4e-47df-a18b-5e251e9e3106-proxy-tls\") pod \"782c0960-aa4e-47df-a18b-5e251e9e3106\" (UID: \"782c0960-aa4e-47df-a18b-5e251e9e3106\") "
Sep 09 21:04:40.028009 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:04:40.027862    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/782c0960-aa4e-47df-a18b-5e251e9e3106-kserve-provision-location\") pod \"782c0960-aa4e-47df-a18b-5e251e9e3106\" (UID: \"782c0960-aa4e-47df-a18b-5e251e9e3106\") "
Sep 09 21:04:40.028229 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:04:40.028204    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/782c0960-aa4e-47df-a18b-5e251e9e3106-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "782c0960-aa4e-47df-a18b-5e251e9e3106" (UID: "782c0960-aa4e-47df-a18b-5e251e9e3106"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:04:40.028229 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:04:40.028212    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/782c0960-aa4e-47df-a18b-5e251e9e3106-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "782c0960-aa4e-47df-a18b-5e251e9e3106" (UID: "782c0960-aa4e-47df-a18b-5e251e9e3106"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:04:40.028315 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:04:40.028240    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/782c0960-aa4e-47df-a18b-5e251e9e3106-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "782c0960-aa4e-47df-a18b-5e251e9e3106" (UID: "782c0960-aa4e-47df-a18b-5e251e9e3106"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:04:40.030222 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:04:40.030196    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/782c0960-aa4e-47df-a18b-5e251e9e3106-kube-api-access-vtsmf" (OuterVolumeSpecName: "kube-api-access-vtsmf") pod "782c0960-aa4e-47df-a18b-5e251e9e3106" (UID: "782c0960-aa4e-47df-a18b-5e251e9e3106"). InnerVolumeSpecName "kube-api-access-vtsmf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:04:40.030286 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:04:40.030219    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/782c0960-aa4e-47df-a18b-5e251e9e3106-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "782c0960-aa4e-47df-a18b-5e251e9e3106" (UID: "782c0960-aa4e-47df-a18b-5e251e9e3106"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:04:40.128459 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:04:40.128393    2579 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/782c0960-aa4e-47df-a18b-5e251e9e3106-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-224.ec2.internal\" DevicePath \"\""
Sep 09 21:04:40.128459 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:04:40.128427    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/782c0960-aa4e-47df-a18b-5e251e9e3106-proxy-tls\") on node \"ip-10-0-138-224.ec2.internal\" DevicePath \"\""
Sep 09 21:04:40.128459 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:04:40.128462    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/782c0960-aa4e-47df-a18b-5e251e9e3106-kserve-provision-location\") on node \"ip-10-0-138-224.ec2.internal\" DevicePath \"\""
Sep 09 21:04:40.128689 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:04:40.128472    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vtsmf\" (UniqueName: \"kubernetes.io/projected/782c0960-aa4e-47df-a18b-5e251e9e3106-kube-api-access-vtsmf\") on node \"ip-10-0-138-224.ec2.internal\" DevicePath \"\""
Sep 09 21:04:40.128689 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:04:40.128483    2579 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/782c0960-aa4e-47df-a18b-5e251e9e3106-cabundle-cert\") on node \"ip-10-0-138-224.ec2.internal\" DevicePath \"\""
Sep 09 21:04:40.559197 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:04:40.559162    2579 generic.go:356] "Generic (PLEG): container finished" podID="782c0960-aa4e-47df-a18b-5e251e9e3106" containerID="3d261d27bca6217b7885a21f98d958fdce3942fa81b7e7a8d791a7bec8ae3ffb" exitCode=0
Sep 09 21:04:40.559370 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:04:40.559209    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-8df9t" event={"ID":"782c0960-aa4e-47df-a18b-5e251e9e3106","Type":"ContainerDied","Data":"3d261d27bca6217b7885a21f98d958fdce3942fa81b7e7a8d791a7bec8ae3ffb"}
Sep 09 21:04:40.559370 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:04:40.559269    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-8df9t" event={"ID":"782c0960-aa4e-47df-a18b-5e251e9e3106","Type":"ContainerDied","Data":"a574b99faa67634c84ab4c97720b31361c4c0f437d28fd7ef6755b8c521ef4c2"}
Sep 09 21:04:40.559370 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:04:40.559220    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-8df9t"
Sep 09 21:04:40.559527 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:04:40.559298    2579 scope.go:117] "RemoveContainer" containerID="fd803a1f8c7ba9421e20b610399b4af991d0ce08bd6b0953cf98761ddabe1658"
Sep 09 21:04:40.568624 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:04:40.568605    2579 scope.go:117] "RemoveContainer" containerID="3d261d27bca6217b7885a21f98d958fdce3942fa81b7e7a8d791a7bec8ae3ffb"
Sep 09 21:04:40.576782 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:04:40.576763    2579 scope.go:117] "RemoveContainer" containerID="fb057de33235899677d846f2fd717e84f2f9b9d2112d96de1c0dbeaf674818f0"
Sep 09 21:04:40.584981 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:04:40.584960    2579 scope.go:117] "RemoveContainer" containerID="fd803a1f8c7ba9421e20b610399b4af991d0ce08bd6b0953cf98761ddabe1658"
Sep 09 21:04:40.585259 ip-10-0-138-224 kubenswrapper[2579]: E0909 21:04:40.585241    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd803a1f8c7ba9421e20b610399b4af991d0ce08bd6b0953cf98761ddabe1658\": container with ID starting with fd803a1f8c7ba9421e20b610399b4af991d0ce08bd6b0953cf98761ddabe1658 not found: ID does not exist" containerID="fd803a1f8c7ba9421e20b610399b4af991d0ce08bd6b0953cf98761ddabe1658"
Sep 09 21:04:40.585313 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:04:40.585268    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd803a1f8c7ba9421e20b610399b4af991d0ce08bd6b0953cf98761ddabe1658"} err="failed to get container status \"fd803a1f8c7ba9421e20b610399b4af991d0ce08bd6b0953cf98761ddabe1658\": rpc error: code = NotFound desc = could not find container \"fd803a1f8c7ba9421e20b610399b4af991d0ce08bd6b0953cf98761ddabe1658\": container with ID starting with fd803a1f8c7ba9421e20b610399b4af991d0ce08bd6b0953cf98761ddabe1658 not found: ID does not exist"
Sep 09 21:04:40.585313 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:04:40.585291    2579 scope.go:117] "RemoveContainer" containerID="3d261d27bca6217b7885a21f98d958fdce3942fa81b7e7a8d791a7bec8ae3ffb"
Sep 09 21:04:40.585609 ip-10-0-138-224 kubenswrapper[2579]: E0909 21:04:40.585586    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d261d27bca6217b7885a21f98d958fdce3942fa81b7e7a8d791a7bec8ae3ffb\": container with ID starting with 3d261d27bca6217b7885a21f98d958fdce3942fa81b7e7a8d791a7bec8ae3ffb not found: ID does not exist" containerID="3d261d27bca6217b7885a21f98d958fdce3942fa81b7e7a8d791a7bec8ae3ffb"
Sep 09 21:04:40.585739 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:04:40.585614    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-8df9t"]
Sep 09 21:04:40.585739 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:04:40.585614    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d261d27bca6217b7885a21f98d958fdce3942fa81b7e7a8d791a7bec8ae3ffb"} err="failed to get container status \"3d261d27bca6217b7885a21f98d958fdce3942fa81b7e7a8d791a7bec8ae3ffb\": rpc error: code = NotFound desc = could not find container \"3d261d27bca6217b7885a21f98d958fdce3942fa81b7e7a8d791a7bec8ae3ffb\": container with ID starting with 3d261d27bca6217b7885a21f98d958fdce3942fa81b7e7a8d791a7bec8ae3ffb not found: ID does not exist"
Sep 09 21:04:40.585739 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:04:40.585666    2579 scope.go:117] "RemoveContainer" containerID="fb057de33235899677d846f2fd717e84f2f9b9d2112d96de1c0dbeaf674818f0"
Sep 09 21:04:40.585946 ip-10-0-138-224 kubenswrapper[2579]: E0909 21:04:40.585927    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb057de33235899677d846f2fd717e84f2f9b9d2112d96de1c0dbeaf674818f0\": container with ID starting with fb057de33235899677d846f2fd717e84f2f9b9d2112d96de1c0dbeaf674818f0 not found: ID does not exist" containerID="fb057de33235899677d846f2fd717e84f2f9b9d2112d96de1c0dbeaf674818f0"
Sep 09 21:04:40.585983 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:04:40.585951    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb057de33235899677d846f2fd717e84f2f9b9d2112d96de1c0dbeaf674818f0"} err="failed to get container status \"fb057de33235899677d846f2fd717e84f2f9b9d2112d96de1c0dbeaf674818f0\": rpc error: code = NotFound desc = could not find container \"fb057de33235899677d846f2fd717e84f2f9b9d2112d96de1c0dbeaf674818f0\": container with ID starting with fb057de33235899677d846f2fd717e84f2f9b9d2112d96de1c0dbeaf674818f0 not found: ID does not exist"
Sep 09 21:04:40.591913 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:04:40.591893    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-8df9t"]
Sep 09 21:04:41.291827 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:04:41.291784    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="782c0960-aa4e-47df-a18b-5e251e9e3106" path="/var/lib/kubelet/pods/782c0960-aa4e-47df-a18b-5e251e9e3106/volumes"
Sep 09 21:04:42.710933 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:04:42.710901    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-vvwdp"]
Sep 09 21:04:42.711295 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:04:42.711247    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="782c0960-aa4e-47df-a18b-5e251e9e3106" containerName="kube-rbac-proxy"
Sep 09 21:04:42.711295 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:04:42.711258    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="782c0960-aa4e-47df-a18b-5e251e9e3106" containerName="kube-rbac-proxy"
Sep 09 21:04:42.711295 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:04:42.711271    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="782c0960-aa4e-47df-a18b-5e251e9e3106" containerName="storage-initializer"
Sep 09 21:04:42.711295 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:04:42.711277    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="782c0960-aa4e-47df-a18b-5e251e9e3106" containerName="storage-initializer"
Sep 09 21:04:42.711295 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:04:42.711287    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="782c0960-aa4e-47df-a18b-5e251e9e3106" containerName="kserve-container"
Sep 09 21:04:42.711295 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:04:42.711293    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="782c0960-aa4e-47df-a18b-5e251e9e3106" containerName="kserve-container"
Sep 09 21:04:42.711495 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:04:42.711341    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="782c0960-aa4e-47df-a18b-5e251e9e3106" containerName="kserve-container"
Sep 09 21:04:42.711495 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:04:42.711350    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="782c0960-aa4e-47df-a18b-5e251e9e3106" containerName="kube-rbac-proxy"
Sep 09 21:04:42.715896 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:04:42.715876    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-vvwdp"
Sep 09 21:04:42.718306 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:04:42.718285    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-predictor-serving-cert\""
Sep 09 21:04:42.718409 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:04:42.718285    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-kube-rbac-proxy-sar-config\""
Sep 09 21:04:42.727185 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:04:42.727163    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-vvwdp"]
Sep 09 21:04:42.850777 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:04:42.850734    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7wrn\" (UniqueName: \"kubernetes.io/projected/340c339e-9ed0-4ba2-b6c0-dad02ef88836-kube-api-access-b7wrn\") pod \"isvc-canary-rollback-predictor-6cb68d45bc-vvwdp\" (UID: \"340c339e-9ed0-4ba2-b6c0-dad02ef88836\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-vvwdp"
Sep 09 21:04:42.850777 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:04:42.850778    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/340c339e-9ed0-4ba2-b6c0-dad02ef88836-cabundle-cert\") pod \"isvc-canary-rollback-predictor-6cb68d45bc-vvwdp\" (UID: \"340c339e-9ed0-4ba2-b6c0-dad02ef88836\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-vvwdp"
Sep 09 21:04:42.851029 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:04:42.850852    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/340c339e-9ed0-4ba2-b6c0-dad02ef88836-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-6cb68d45bc-vvwdp\" (UID: \"340c339e-9ed0-4ba2-b6c0-dad02ef88836\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-vvwdp"
Sep 09 21:04:42.851029 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:04:42.850911    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/340c339e-9ed0-4ba2-b6c0-dad02ef88836-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-6cb68d45bc-vvwdp\" (UID: \"340c339e-9ed0-4ba2-b6c0-dad02ef88836\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-vvwdp"
Sep 09 21:04:42.851029 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:04:42.850946    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/340c339e-9ed0-4ba2-b6c0-dad02ef88836-proxy-tls\") pod \"isvc-canary-rollback-predictor-6cb68d45bc-vvwdp\" (UID: \"340c339e-9ed0-4ba2-b6c0-dad02ef88836\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-vvwdp"
Sep 09 21:04:42.952142 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:04:42.952094    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/340c339e-9ed0-4ba2-b6c0-dad02ef88836-proxy-tls\") pod \"isvc-canary-rollback-predictor-6cb68d45bc-vvwdp\" (UID: \"340c339e-9ed0-4ba2-b6c0-dad02ef88836\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-vvwdp"
Sep 09 21:04:42.952336 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:04:42.952181    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7wrn\" (UniqueName: \"kubernetes.io/projected/340c339e-9ed0-4ba2-b6c0-dad02ef88836-kube-api-access-b7wrn\") pod \"isvc-canary-rollback-predictor-6cb68d45bc-vvwdp\" (UID: \"340c339e-9ed0-4ba2-b6c0-dad02ef88836\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-vvwdp"
Sep 09 21:04:42.952336 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:04:42.952203    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/340c339e-9ed0-4ba2-b6c0-dad02ef88836-cabundle-cert\") pod \"isvc-canary-rollback-predictor-6cb68d45bc-vvwdp\" (UID: \"340c339e-9ed0-4ba2-b6c0-dad02ef88836\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-vvwdp"
Sep 09 21:04:42.952336 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:04:42.952238    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/340c339e-9ed0-4ba2-b6c0-dad02ef88836-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-6cb68d45bc-vvwdp\" (UID: \"340c339e-9ed0-4ba2-b6c0-dad02ef88836\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-vvwdp"
Sep 09 21:04:42.952336 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:04:42.952261    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/340c339e-9ed0-4ba2-b6c0-dad02ef88836-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-6cb68d45bc-vvwdp\" (UID: \"340c339e-9ed0-4ba2-b6c0-dad02ef88836\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-vvwdp"
Sep 09 21:04:42.952792 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:04:42.952759    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/340c339e-9ed0-4ba2-b6c0-dad02ef88836-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-6cb68d45bc-vvwdp\" (UID: \"340c339e-9ed0-4ba2-b6c0-dad02ef88836\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-vvwdp"
Sep 09 21:04:42.952969 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:04:42.952952    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/340c339e-9ed0-4ba2-b6c0-dad02ef88836-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-6cb68d45bc-vvwdp\" (UID: \"340c339e-9ed0-4ba2-b6c0-dad02ef88836\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-vvwdp"
Sep 09 21:04:42.953015 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:04:42.952987    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/340c339e-9ed0-4ba2-b6c0-dad02ef88836-cabundle-cert\") pod \"isvc-canary-rollback-predictor-6cb68d45bc-vvwdp\" (UID: \"340c339e-9ed0-4ba2-b6c0-dad02ef88836\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-vvwdp"
Sep 09 21:04:42.954804 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:04:42.954783    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/340c339e-9ed0-4ba2-b6c0-dad02ef88836-proxy-tls\") pod \"isvc-canary-rollback-predictor-6cb68d45bc-vvwdp\" (UID: \"340c339e-9ed0-4ba2-b6c0-dad02ef88836\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-vvwdp"
Sep 09 21:04:42.960543 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:04:42.960523    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7wrn\" (UniqueName: \"kubernetes.io/projected/340c339e-9ed0-4ba2-b6c0-dad02ef88836-kube-api-access-b7wrn\") pod \"isvc-canary-rollback-predictor-6cb68d45bc-vvwdp\" (UID: \"340c339e-9ed0-4ba2-b6c0-dad02ef88836\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-vvwdp"
Sep 09 21:04:43.026467 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:04:43.026365    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-vvwdp"
Sep 09 21:04:43.302758 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:04:43.302733    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-vvwdp"]
Sep 09 21:04:43.304867 ip-10-0-138-224 kubenswrapper[2579]: W0909 21:04:43.304842    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod340c339e_9ed0_4ba2_b6c0_dad02ef88836.slice/crio-16252c06e44d1cf10f4f8ec51f96661128d77aa15a63bbbd5bd1eab075d9c352 WatchSource:0}: Error finding container 16252c06e44d1cf10f4f8ec51f96661128d77aa15a63bbbd5bd1eab075d9c352: Status 404 returned error can't find the container with id 16252c06e44d1cf10f4f8ec51f96661128d77aa15a63bbbd5bd1eab075d9c352
Sep 09 21:04:43.576958 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:04:43.576850    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-vvwdp" event={"ID":"340c339e-9ed0-4ba2-b6c0-dad02ef88836","Type":"ContainerStarted","Data":"fbc6b7d7f58e14e84766c0e49905b0d352c32089b65260c9a3465c9e4408482e"}
Sep 09 21:04:43.577113 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:04:43.576972    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-vvwdp" event={"ID":"340c339e-9ed0-4ba2-b6c0-dad02ef88836","Type":"ContainerStarted","Data":"16252c06e44d1cf10f4f8ec51f96661128d77aa15a63bbbd5bd1eab075d9c352"}
Sep 09 21:04:47.596465 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:04:47.596400    2579 generic.go:356] "Generic (PLEG): container finished" podID="340c339e-9ed0-4ba2-b6c0-dad02ef88836" containerID="fbc6b7d7f58e14e84766c0e49905b0d352c32089b65260c9a3465c9e4408482e" exitCode=0
Sep 09 21:04:47.596465 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:04:47.596467    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-vvwdp" event={"ID":"340c339e-9ed0-4ba2-b6c0-dad02ef88836","Type":"ContainerDied","Data":"fbc6b7d7f58e14e84766c0e49905b0d352c32089b65260c9a3465c9e4408482e"}
Sep 09 21:04:47.771564 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:04:47.771523    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2a6f9-predictor-86589975f4-jprmq"]
Sep 09 21:04:47.771850 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:04:47.771820    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2a6f9-predictor-86589975f4-jprmq" podUID="c9e563c3-c2c7-44c1-8061-f7730429b4ee" containerName="kserve-container" containerID="cri-o://5bf007785187b3f23ae9ffb8f69b669c4f8b9b04a6816ab3faa31ad35ef7c8d7" gracePeriod=30
Sep 09 21:04:47.771997 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:04:47.771841    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2a6f9-predictor-86589975f4-jprmq" podUID="c9e563c3-c2c7-44c1-8061-f7730429b4ee" containerName="agent" containerID="cri-o://9f52b3f4b8b5a088dae748eb8d7f33245ba2a46f936572e7a33aaeca5881579e" gracePeriod=30
Sep 09 21:04:47.771997 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:04:47.771850    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2a6f9-predictor-86589975f4-jprmq" podUID="c9e563c3-c2c7-44c1-8061-f7730429b4ee" containerName="kube-rbac-proxy" containerID="cri-o://45c3765997542ba42dd0dcbf0fddf8a6c1e09718453cc85f2d2b907852304d27" gracePeriod=30
Sep 09 21:04:48.235942 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:04:48.235894    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2a6f9-predictor-86589975f4-jprmq" podUID="c9e563c3-c2c7-44c1-8061-f7730429b4ee" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 09 21:04:48.236272 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:04:48.236241    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2a6f9-predictor-86589975f4-jprmq" podUID="c9e563c3-c2c7-44c1-8061-f7730429b4ee" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 21:04:48.603390 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:04:48.603307    2579 generic.go:356] "Generic (PLEG): container finished" podID="c9e563c3-c2c7-44c1-8061-f7730429b4ee" containerID="45c3765997542ba42dd0dcbf0fddf8a6c1e09718453cc85f2d2b907852304d27" exitCode=2
Sep 09 21:04:48.603790 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:04:48.603385    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2a6f9-predictor-86589975f4-jprmq" event={"ID":"c9e563c3-c2c7-44c1-8061-f7730429b4ee","Type":"ContainerDied","Data":"45c3765997542ba42dd0dcbf0fddf8a6c1e09718453cc85f2d2b907852304d27"}
Sep 09 21:04:48.605556 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:04:48.605535    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-vvwdp" event={"ID":"340c339e-9ed0-4ba2-b6c0-dad02ef88836","Type":"ContainerStarted","Data":"5af29832417a139c8fdf8ebc8044f9f8226c7481a4cb279a24515e66e6ad2c9e"}
Sep 09 21:04:48.605642 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:04:48.605580    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-vvwdp" event={"ID":"340c339e-9ed0-4ba2-b6c0-dad02ef88836","Type":"ContainerStarted","Data":"cf9837d9938536eb2a2a9463b646dffbc1a343d9243e31c64fa372c2dab47c7e"}
Sep 09 21:04:48.605778 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:04:48.605763    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-vvwdp"
Sep 09 21:04:49.610050 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:04:49.610016    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-vvwdp"
Sep 09 21:04:49.611430 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:04:49.611396    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-vvwdp" podUID="340c339e-9ed0-4ba2-b6c0-dad02ef88836" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 09 21:04:50.615373 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:04:50.615334    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-vvwdp" podUID="340c339e-9ed0-4ba2-b6c0-dad02ef88836" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 09 21:04:51.224735 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:04:51.224685    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2a6f9-predictor-86589975f4-jprmq" podUID="c9e563c3-c2c7-44c1-8061-f7730429b4ee" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.31:8643/healthz\": dial tcp 10.134.0.31:8643: connect: connection refused"
Sep 09 21:04:53.631665 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:04:53.631626    2579 generic.go:356] "Generic (PLEG): container finished" podID="c9e563c3-c2c7-44c1-8061-f7730429b4ee" containerID="5bf007785187b3f23ae9ffb8f69b669c4f8b9b04a6816ab3faa31ad35ef7c8d7" exitCode=0
Sep 09 21:04:53.632026 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:04:53.631694    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2a6f9-predictor-86589975f4-jprmq" event={"ID":"c9e563c3-c2c7-44c1-8061-f7730429b4ee","Type":"ContainerDied","Data":"5bf007785187b3f23ae9ffb8f69b669c4f8b9b04a6816ab3faa31ad35ef7c8d7"}
Sep 09 21:04:55.620722 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:04:55.620693    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-vvwdp"
Sep 09 21:04:55.621282 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:04:55.621255    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-vvwdp" podUID="340c339e-9ed0-4ba2-b6c0-dad02ef88836" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 09 21:04:55.643684 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:04:55.643633    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-vvwdp" podStartSLOduration=9.944664587 podStartE2EDuration="13.643618662s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.698954075s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:04:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:04:48.646708256 +0000 UTC m=+745.886802374" watchObservedRunningTime="2026-09-09 21:04:55.643618662 +0000 UTC m=+752.883712780"
Sep 09 21:04:56.224297 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:04:56.224253    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2a6f9-predictor-86589975f4-jprmq" podUID="c9e563c3-c2c7-44c1-8061-f7730429b4ee" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.31:8643/healthz\": dial tcp 10.134.0.31:8643: connect: connection refused"
Sep 09 21:04:58.235924 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:04:58.235869    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2a6f9-predictor-86589975f4-jprmq" podUID="c9e563c3-c2c7-44c1-8061-f7730429b4ee" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 09 21:04:58.236356 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:04:58.236222    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2a6f9-predictor-86589975f4-jprmq" podUID="c9e563c3-c2c7-44c1-8061-f7730429b4ee" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 21:05:01.224371 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:05:01.224329    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2a6f9-predictor-86589975f4-jprmq" podUID="c9e563c3-c2c7-44c1-8061-f7730429b4ee" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.31:8643/healthz\": dial tcp 10.134.0.31:8643: connect: connection refused"
Sep 09 21:05:01.224866 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:05:01.224509    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2a6f9-predictor-86589975f4-jprmq"
Sep 09 21:05:05.621170 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:05:05.621125    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-vvwdp" podUID="340c339e-9ed0-4ba2-b6c0-dad02ef88836" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 09 21:05:06.224085 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:05:06.224042    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2a6f9-predictor-86589975f4-jprmq" podUID="c9e563c3-c2c7-44c1-8061-f7730429b4ee" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.31:8643/healthz\": dial tcp 10.134.0.31:8643: connect: connection refused"
Sep 09 21:05:08.235682 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:05:08.235614    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2a6f9-predictor-86589975f4-jprmq" podUID="c9e563c3-c2c7-44c1-8061-f7730429b4ee" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 09 21:05:08.236154 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:05:08.235828    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2a6f9-predictor-86589975f4-jprmq"
Sep 09 21:05:08.236154 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:05:08.235973    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2a6f9-predictor-86589975f4-jprmq" podUID="c9e563c3-c2c7-44c1-8061-f7730429b4ee" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 21:05:08.236154 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:05:08.236082    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2a6f9-predictor-86589975f4-jprmq"
Sep 09 21:05:11.224297 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:05:11.224251    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2a6f9-predictor-86589975f4-jprmq" podUID="c9e563c3-c2c7-44c1-8061-f7730429b4ee" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.31:8643/healthz\": dial tcp 10.134.0.31:8643: connect: connection refused"
Sep 09 21:05:15.621969 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:05:15.621920    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-vvwdp" podUID="340c339e-9ed0-4ba2-b6c0-dad02ef88836" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 09 21:05:16.224192 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:05:16.224153    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2a6f9-predictor-86589975f4-jprmq" podUID="c9e563c3-c2c7-44c1-8061-f7730429b4ee" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.31:8643/healthz\": dial tcp 10.134.0.31:8643: connect: connection refused"
Sep 09 21:05:17.923176 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:05:17.923151    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2a6f9-predictor-86589975f4-jprmq"
Sep 09 21:05:17.958731 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:05:17.958705    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c9e563c3-c2c7-44c1-8061-f7730429b4ee-kube-rbac-proxy-sar-config\") pod \"c9e563c3-c2c7-44c1-8061-f7730429b4ee\" (UID: \"c9e563c3-c2c7-44c1-8061-f7730429b4ee\") "
Sep 09 21:05:17.958857 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:05:17.958772    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c9e563c3-c2c7-44c1-8061-f7730429b4ee-proxy-tls\") pod \"c9e563c3-c2c7-44c1-8061-f7730429b4ee\" (UID: \"c9e563c3-c2c7-44c1-8061-f7730429b4ee\") "
Sep 09 21:05:17.958857 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:05:17.958826    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c9e563c3-c2c7-44c1-8061-f7730429b4ee-cabundle-cert\") pod \"c9e563c3-c2c7-44c1-8061-f7730429b4ee\" (UID: \"c9e563c3-c2c7-44c1-8061-f7730429b4ee\") "
Sep 09 21:05:17.958976 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:05:17.958858    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sf979\" (UniqueName: \"kubernetes.io/projected/c9e563c3-c2c7-44c1-8061-f7730429b4ee-kube-api-access-sf979\") pod \"c9e563c3-c2c7-44c1-8061-f7730429b4ee\" (UID: \"c9e563c3-c2c7-44c1-8061-f7730429b4ee\") "
Sep 09 21:05:17.958976 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:05:17.958910    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c9e563c3-c2c7-44c1-8061-f7730429b4ee-kserve-provision-location\") pod \"c9e563c3-c2c7-44c1-8061-f7730429b4ee\" (UID: \"c9e563c3-c2c7-44c1-8061-f7730429b4ee\") "
Sep 09 21:05:17.959090 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:05:17.959058    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9e563c3-c2c7-44c1-8061-f7730429b4ee-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "c9e563c3-c2c7-44c1-8061-f7730429b4ee" (UID: "c9e563c3-c2c7-44c1-8061-f7730429b4ee"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:05:17.959212 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:05:17.959177    2579 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c9e563c3-c2c7-44c1-8061-f7730429b4ee-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-224.ec2.internal\" DevicePath \"\""
Sep 09 21:05:17.959312 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:05:17.959173    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9e563c3-c2c7-44c1-8061-f7730429b4ee-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "c9e563c3-c2c7-44c1-8061-f7730429b4ee" (UID: "c9e563c3-c2c7-44c1-8061-f7730429b4ee"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:05:17.959396 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:05:17.959316    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9e563c3-c2c7-44c1-8061-f7730429b4ee-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c9e563c3-c2c7-44c1-8061-f7730429b4ee" (UID: "c9e563c3-c2c7-44c1-8061-f7730429b4ee"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:05:17.960968 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:05:17.960949    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9e563c3-c2c7-44c1-8061-f7730429b4ee-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c9e563c3-c2c7-44c1-8061-f7730429b4ee" (UID: "c9e563c3-c2c7-44c1-8061-f7730429b4ee"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:05:17.961122 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:05:17.961108    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9e563c3-c2c7-44c1-8061-f7730429b4ee-kube-api-access-sf979" (OuterVolumeSpecName: "kube-api-access-sf979") pod "c9e563c3-c2c7-44c1-8061-f7730429b4ee" (UID: "c9e563c3-c2c7-44c1-8061-f7730429b4ee"). InnerVolumeSpecName "kube-api-access-sf979". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:05:18.060146 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:05:18.060089    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c9e563c3-c2c7-44c1-8061-f7730429b4ee-proxy-tls\") on node \"ip-10-0-138-224.ec2.internal\" DevicePath \"\""
Sep 09 21:05:18.060146 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:05:18.060114    2579 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c9e563c3-c2c7-44c1-8061-f7730429b4ee-cabundle-cert\") on node \"ip-10-0-138-224.ec2.internal\" DevicePath \"\""
Sep 09 21:05:18.060146 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:05:18.060124    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sf979\" (UniqueName: \"kubernetes.io/projected/c9e563c3-c2c7-44c1-8061-f7730429b4ee-kube-api-access-sf979\") on node \"ip-10-0-138-224.ec2.internal\" DevicePath \"\""
Sep 09 21:05:18.060146 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:05:18.060135    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c9e563c3-c2c7-44c1-8061-f7730429b4ee-kserve-provision-location\") on node \"ip-10-0-138-224.ec2.internal\" DevicePath \"\""
Sep 09 21:05:18.748902 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:05:18.748867    2579 generic.go:356] "Generic (PLEG): container finished" podID="c9e563c3-c2c7-44c1-8061-f7730429b4ee" containerID="9f52b3f4b8b5a088dae748eb8d7f33245ba2a46f936572e7a33aaeca5881579e" exitCode=0
Sep 09 21:05:18.749075 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:05:18.748930    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2a6f9-predictor-86589975f4-jprmq"
Sep 09 21:05:18.749075 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:05:18.748923    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2a6f9-predictor-86589975f4-jprmq" event={"ID":"c9e563c3-c2c7-44c1-8061-f7730429b4ee","Type":"ContainerDied","Data":"9f52b3f4b8b5a088dae748eb8d7f33245ba2a46f936572e7a33aaeca5881579e"}
Sep 09 21:05:18.749075 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:05:18.749063    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2a6f9-predictor-86589975f4-jprmq" event={"ID":"c9e563c3-c2c7-44c1-8061-f7730429b4ee","Type":"ContainerDied","Data":"c1f4f95834a53a2272452e3c362c9f2e57d2ad9a46f4f3219c4da6c17fb525d7"}
Sep 09 21:05:18.749193 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:05:18.749095    2579 scope.go:117] "RemoveContainer" containerID="9f52b3f4b8b5a088dae748eb8d7f33245ba2a46f936572e7a33aaeca5881579e"
Sep 09 21:05:18.758151 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:05:18.758133    2579 scope.go:117] "RemoveContainer" containerID="45c3765997542ba42dd0dcbf0fddf8a6c1e09718453cc85f2d2b907852304d27"
Sep 09 21:05:18.765851 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:05:18.765832    2579 scope.go:117] "RemoveContainer" containerID="5bf007785187b3f23ae9ffb8f69b669c4f8b9b04a6816ab3faa31ad35ef7c8d7"
Sep 09 21:05:18.772420 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:05:18.772400    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2a6f9-predictor-86589975f4-jprmq"]
Sep 09 21:05:18.773288 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:05:18.773272    2579 scope.go:117] "RemoveContainer" containerID="3fae88b1cd6d0d8508909885184965fef5a4f35d1aed08c7dd1a4ec45c29e6b0"
Sep 09 21:05:18.776111 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:05:18.776089    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2a6f9-predictor-86589975f4-jprmq"]
Sep 09 21:05:18.782561 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:05:18.782378    2579 scope.go:117] "RemoveContainer" containerID="9f52b3f4b8b5a088dae748eb8d7f33245ba2a46f936572e7a33aaeca5881579e"
Sep 09 21:05:18.782741 ip-10-0-138-224 kubenswrapper[2579]: E0909 21:05:18.782711    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f52b3f4b8b5a088dae748eb8d7f33245ba2a46f936572e7a33aaeca5881579e\": container with ID starting with 9f52b3f4b8b5a088dae748eb8d7f33245ba2a46f936572e7a33aaeca5881579e not found: ID does not exist" containerID="9f52b3f4b8b5a088dae748eb8d7f33245ba2a46f936572e7a33aaeca5881579e"
Sep 09 21:05:18.782790 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:05:18.782750    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f52b3f4b8b5a088dae748eb8d7f33245ba2a46f936572e7a33aaeca5881579e"} err="failed to get container status \"9f52b3f4b8b5a088dae748eb8d7f33245ba2a46f936572e7a33aaeca5881579e\": rpc error: code = NotFound desc = could not find container \"9f52b3f4b8b5a088dae748eb8d7f33245ba2a46f936572e7a33aaeca5881579e\": container with ID starting with 9f52b3f4b8b5a088dae748eb8d7f33245ba2a46f936572e7a33aaeca5881579e not found: ID does not exist"
Sep 09 21:05:18.782790 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:05:18.782767    2579 scope.go:117] "RemoveContainer" containerID="45c3765997542ba42dd0dcbf0fddf8a6c1e09718453cc85f2d2b907852304d27"
Sep 09 21:05:18.783065 ip-10-0-138-224 kubenswrapper[2579]: E0909 21:05:18.783049    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45c3765997542ba42dd0dcbf0fddf8a6c1e09718453cc85f2d2b907852304d27\": container with ID starting with 45c3765997542ba42dd0dcbf0fddf8a6c1e09718453cc85f2d2b907852304d27 not found: ID does not exist" containerID="45c3765997542ba42dd0dcbf0fddf8a6c1e09718453cc85f2d2b907852304d27"
Sep 09 21:05:18.783107 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:05:18.783071    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45c3765997542ba42dd0dcbf0fddf8a6c1e09718453cc85f2d2b907852304d27"} err="failed to get container status \"45c3765997542ba42dd0dcbf0fddf8a6c1e09718453cc85f2d2b907852304d27\": rpc error: code = NotFound desc = could not find container \"45c3765997542ba42dd0dcbf0fddf8a6c1e09718453cc85f2d2b907852304d27\": container with ID starting with 45c3765997542ba42dd0dcbf0fddf8a6c1e09718453cc85f2d2b907852304d27 not found: ID does not exist"
Sep 09 21:05:18.783107 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:05:18.783087    2579 scope.go:117] "RemoveContainer" containerID="5bf007785187b3f23ae9ffb8f69b669c4f8b9b04a6816ab3faa31ad35ef7c8d7"
Sep 09 21:05:18.783327 ip-10-0-138-224 kubenswrapper[2579]: E0909 21:05:18.783312    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5bf007785187b3f23ae9ffb8f69b669c4f8b9b04a6816ab3faa31ad35ef7c8d7\": container with ID starting with 5bf007785187b3f23ae9ffb8f69b669c4f8b9b04a6816ab3faa31ad35ef7c8d7 not found: ID does not exist" containerID="5bf007785187b3f23ae9ffb8f69b669c4f8b9b04a6816ab3faa31ad35ef7c8d7"
Sep 09 21:05:18.783367 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:05:18.783340    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5bf007785187b3f23ae9ffb8f69b669c4f8b9b04a6816ab3faa31ad35ef7c8d7"} err="failed to get container status \"5bf007785187b3f23ae9ffb8f69b669c4f8b9b04a6816ab3faa31ad35ef7c8d7\": rpc error: code = NotFound desc = could not find container \"5bf007785187b3f23ae9ffb8f69b669c4f8b9b04a6816ab3faa31ad35ef7c8d7\": container with ID starting with 5bf007785187b3f23ae9ffb8f69b669c4f8b9b04a6816ab3faa31ad35ef7c8d7 not found: ID does not exist"
Sep 09 21:05:18.783367 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:05:18.783357    2579 scope.go:117] "RemoveContainer" containerID="3fae88b1cd6d0d8508909885184965fef5a4f35d1aed08c7dd1a4ec45c29e6b0"
Sep 09 21:05:18.783656 ip-10-0-138-224 kubenswrapper[2579]: E0909 21:05:18.783638    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3fae88b1cd6d0d8508909885184965fef5a4f35d1aed08c7dd1a4ec45c29e6b0\": container with ID starting with 3fae88b1cd6d0d8508909885184965fef5a4f35d1aed08c7dd1a4ec45c29e6b0 not found: ID does not exist" containerID="3fae88b1cd6d0d8508909885184965fef5a4f35d1aed08c7dd1a4ec45c29e6b0"
Sep 09 21:05:18.783723 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:05:18.783664    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3fae88b1cd6d0d8508909885184965fef5a4f35d1aed08c7dd1a4ec45c29e6b0"} err="failed to get container status \"3fae88b1cd6d0d8508909885184965fef5a4f35d1aed08c7dd1a4ec45c29e6b0\": rpc error: code = NotFound desc = could not find container \"3fae88b1cd6d0d8508909885184965fef5a4f35d1aed08c7dd1a4ec45c29e6b0\": container with ID starting with 3fae88b1cd6d0d8508909885184965fef5a4f35d1aed08c7dd1a4ec45c29e6b0 not found: ID does not exist"
Sep 09 21:05:19.291422 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:05:19.291388    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c9e563c3-c2c7-44c1-8061-f7730429b4ee" path="/var/lib/kubelet/pods/c9e563c3-c2c7-44c1-8061-f7730429b4ee/volumes"
Sep 09 21:05:25.621203 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:05:25.621160    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-vvwdp" podUID="340c339e-9ed0-4ba2-b6c0-dad02ef88836" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 09 21:05:35.621605 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:05:35.621560    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-vvwdp" podUID="340c339e-9ed0-4ba2-b6c0-dad02ef88836" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 09 21:05:45.621881 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:05:45.621792    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-vvwdp" podUID="340c339e-9ed0-4ba2-b6c0-dad02ef88836" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 09 21:05:55.621718 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:05:55.621673    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-vvwdp" podUID="340c339e-9ed0-4ba2-b6c0-dad02ef88836" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 09 21:06:05.622617 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:06:05.622583    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-vvwdp"
Sep 09 21:06:12.764762 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:06:12.764727    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-vvwdp"]
Sep 09 21:06:12.765320 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:06:12.765025    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-vvwdp" podUID="340c339e-9ed0-4ba2-b6c0-dad02ef88836" containerName="kserve-container" containerID="cri-o://cf9837d9938536eb2a2a9463b646dffbc1a343d9243e31c64fa372c2dab47c7e" gracePeriod=30
Sep 09 21:06:12.765320 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:06:12.765087    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-vvwdp" podUID="340c339e-9ed0-4ba2-b6c0-dad02ef88836" containerName="kube-rbac-proxy" containerID="cri-o://5af29832417a139c8fdf8ebc8044f9f8226c7481a4cb279a24515e66e6ad2c9e" gracePeriod=30
Sep 09 21:06:12.989620 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:06:12.989593    2579 generic.go:356] "Generic (PLEG): container finished" podID="340c339e-9ed0-4ba2-b6c0-dad02ef88836" containerID="5af29832417a139c8fdf8ebc8044f9f8226c7481a4cb279a24515e66e6ad2c9e" exitCode=2
Sep 09 21:06:12.989727 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:06:12.989665    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-vvwdp" event={"ID":"340c339e-9ed0-4ba2-b6c0-dad02ef88836","Type":"ContainerDied","Data":"5af29832417a139c8fdf8ebc8044f9f8226c7481a4cb279a24515e66e6ad2c9e"}
Sep 09 21:06:15.616369 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:06:15.616331    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-vvwdp" podUID="340c339e-9ed0-4ba2-b6c0-dad02ef88836" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.32:8643/healthz\": dial tcp 10.134.0.32:8643: connect: connection refused"
Sep 09 21:06:15.621757 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:06:15.621732    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-vvwdp" podUID="340c339e-9ed0-4ba2-b6c0-dad02ef88836" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 09 21:06:16.799613 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:06:16.799588    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-vvwdp"
Sep 09 21:06:16.825631 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:06:16.825609    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/340c339e-9ed0-4ba2-b6c0-dad02ef88836-kube-rbac-proxy-sar-config\") pod \"340c339e-9ed0-4ba2-b6c0-dad02ef88836\" (UID: \"340c339e-9ed0-4ba2-b6c0-dad02ef88836\") "
Sep 09 21:06:16.825758 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:06:16.825641    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/340c339e-9ed0-4ba2-b6c0-dad02ef88836-cabundle-cert\") pod \"340c339e-9ed0-4ba2-b6c0-dad02ef88836\" (UID: \"340c339e-9ed0-4ba2-b6c0-dad02ef88836\") "
Sep 09 21:06:16.825758 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:06:16.825658    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/340c339e-9ed0-4ba2-b6c0-dad02ef88836-kserve-provision-location\") pod \"340c339e-9ed0-4ba2-b6c0-dad02ef88836\" (UID: \"340c339e-9ed0-4ba2-b6c0-dad02ef88836\") "
Sep 09 21:06:16.825873 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:06:16.825809    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/340c339e-9ed0-4ba2-b6c0-dad02ef88836-proxy-tls\") pod \"340c339e-9ed0-4ba2-b6c0-dad02ef88836\" (UID: \"340c339e-9ed0-4ba2-b6c0-dad02ef88836\") "
Sep 09 21:06:16.825873 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:06:16.825861    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b7wrn\" (UniqueName: \"kubernetes.io/projected/340c339e-9ed0-4ba2-b6c0-dad02ef88836-kube-api-access-b7wrn\") pod \"340c339e-9ed0-4ba2-b6c0-dad02ef88836\" (UID: \"340c339e-9ed0-4ba2-b6c0-dad02ef88836\") "
Sep 09 21:06:16.825985 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:06:16.825934    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/340c339e-9ed0-4ba2-b6c0-dad02ef88836-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "340c339e-9ed0-4ba2-b6c0-dad02ef88836" (UID: "340c339e-9ed0-4ba2-b6c0-dad02ef88836"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:06:16.825985 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:06:16.825968    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/340c339e-9ed0-4ba2-b6c0-dad02ef88836-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "340c339e-9ed0-4ba2-b6c0-dad02ef88836" (UID: "340c339e-9ed0-4ba2-b6c0-dad02ef88836"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:06:16.826084 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:06:16.825995    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/340c339e-9ed0-4ba2-b6c0-dad02ef88836-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "340c339e-9ed0-4ba2-b6c0-dad02ef88836" (UID: "340c339e-9ed0-4ba2-b6c0-dad02ef88836"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:06:16.826188 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:06:16.826171    2579 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/340c339e-9ed0-4ba2-b6c0-dad02ef88836-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-224.ec2.internal\" DevicePath \"\""
Sep 09 21:06:16.826243 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:06:16.826195    2579 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/340c339e-9ed0-4ba2-b6c0-dad02ef88836-cabundle-cert\") on node \"ip-10-0-138-224.ec2.internal\" DevicePath \"\""
Sep 09 21:06:16.826243 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:06:16.826210    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/340c339e-9ed0-4ba2-b6c0-dad02ef88836-kserve-provision-location\") on node \"ip-10-0-138-224.ec2.internal\" DevicePath \"\""
Sep 09 21:06:16.827827 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:06:16.827808    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/340c339e-9ed0-4ba2-b6c0-dad02ef88836-kube-api-access-b7wrn" (OuterVolumeSpecName: "kube-api-access-b7wrn") pod "340c339e-9ed0-4ba2-b6c0-dad02ef88836" (UID: "340c339e-9ed0-4ba2-b6c0-dad02ef88836"). InnerVolumeSpecName "kube-api-access-b7wrn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:06:16.827922 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:06:16.827903    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/340c339e-9ed0-4ba2-b6c0-dad02ef88836-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "340c339e-9ed0-4ba2-b6c0-dad02ef88836" (UID: "340c339e-9ed0-4ba2-b6c0-dad02ef88836"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:06:16.927487 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:06:16.927467    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/340c339e-9ed0-4ba2-b6c0-dad02ef88836-proxy-tls\") on node \"ip-10-0-138-224.ec2.internal\" DevicePath \"\""
Sep 09 21:06:16.927487 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:06:16.927486    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b7wrn\" (UniqueName: \"kubernetes.io/projected/340c339e-9ed0-4ba2-b6c0-dad02ef88836-kube-api-access-b7wrn\") on node \"ip-10-0-138-224.ec2.internal\" DevicePath \"\""
Sep 09 21:06:17.008960 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:06:17.008933    2579 generic.go:356] "Generic (PLEG): container finished" podID="340c339e-9ed0-4ba2-b6c0-dad02ef88836" containerID="cf9837d9938536eb2a2a9463b646dffbc1a343d9243e31c64fa372c2dab47c7e" exitCode=0
Sep 09 21:06:17.009071 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:06:17.009001    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-vvwdp"
Sep 09 21:06:17.009071 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:06:17.009015    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-vvwdp" event={"ID":"340c339e-9ed0-4ba2-b6c0-dad02ef88836","Type":"ContainerDied","Data":"cf9837d9938536eb2a2a9463b646dffbc1a343d9243e31c64fa372c2dab47c7e"}
Sep 09 21:06:17.009147 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:06:17.009096    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-vvwdp" event={"ID":"340c339e-9ed0-4ba2-b6c0-dad02ef88836","Type":"ContainerDied","Data":"16252c06e44d1cf10f4f8ec51f96661128d77aa15a63bbbd5bd1eab075d9c352"}
Sep 09 21:06:17.009147 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:06:17.009123    2579 scope.go:117] "RemoveContainer" containerID="5af29832417a139c8fdf8ebc8044f9f8226c7481a4cb279a24515e66e6ad2c9e"
Sep 09 21:06:17.019742 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:06:17.019721    2579 scope.go:117] "RemoveContainer" containerID="cf9837d9938536eb2a2a9463b646dffbc1a343d9243e31c64fa372c2dab47c7e"
Sep 09 21:06:17.027100 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:06:17.027082    2579 scope.go:117] "RemoveContainer" containerID="fbc6b7d7f58e14e84766c0e49905b0d352c32089b65260c9a3465c9e4408482e"
Sep 09 21:06:17.034339 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:06:17.034284    2579 scope.go:117] "RemoveContainer" containerID="5af29832417a139c8fdf8ebc8044f9f8226c7481a4cb279a24515e66e6ad2c9e"
Sep 09 21:06:17.034647 ip-10-0-138-224 kubenswrapper[2579]: E0909 21:06:17.034617    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5af29832417a139c8fdf8ebc8044f9f8226c7481a4cb279a24515e66e6ad2c9e\": container with ID starting with 5af29832417a139c8fdf8ebc8044f9f8226c7481a4cb279a24515e66e6ad2c9e not found: ID does not exist" containerID="5af29832417a139c8fdf8ebc8044f9f8226c7481a4cb279a24515e66e6ad2c9e"
Sep 09 21:06:17.034795 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:06:17.034654    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5af29832417a139c8fdf8ebc8044f9f8226c7481a4cb279a24515e66e6ad2c9e"} err="failed to get container status \"5af29832417a139c8fdf8ebc8044f9f8226c7481a4cb279a24515e66e6ad2c9e\": rpc error: code = NotFound desc = could not find container \"5af29832417a139c8fdf8ebc8044f9f8226c7481a4cb279a24515e66e6ad2c9e\": container with ID starting with 5af29832417a139c8fdf8ebc8044f9f8226c7481a4cb279a24515e66e6ad2c9e not found: ID does not exist"
Sep 09 21:06:17.034795 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:06:17.034702    2579 scope.go:117] "RemoveContainer" containerID="cf9837d9938536eb2a2a9463b646dffbc1a343d9243e31c64fa372c2dab47c7e"
Sep 09 21:06:17.035085 ip-10-0-138-224 kubenswrapper[2579]: E0909 21:06:17.035041    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf9837d9938536eb2a2a9463b646dffbc1a343d9243e31c64fa372c2dab47c7e\": container with ID starting with cf9837d9938536eb2a2a9463b646dffbc1a343d9243e31c64fa372c2dab47c7e not found: ID does not exist" containerID="cf9837d9938536eb2a2a9463b646dffbc1a343d9243e31c64fa372c2dab47c7e"
Sep 09 21:06:17.035169 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:06:17.035091    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf9837d9938536eb2a2a9463b646dffbc1a343d9243e31c64fa372c2dab47c7e"} err="failed to get container status \"cf9837d9938536eb2a2a9463b646dffbc1a343d9243e31c64fa372c2dab47c7e\": rpc error: code = NotFound desc = could not find container \"cf9837d9938536eb2a2a9463b646dffbc1a343d9243e31c64fa372c2dab47c7e\": container with ID starting with cf9837d9938536eb2a2a9463b646dffbc1a343d9243e31c64fa372c2dab47c7e not found: ID does not exist"
Sep 09 21:06:17.035169 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:06:17.035112    2579 scope.go:117] "RemoveContainer" containerID="fbc6b7d7f58e14e84766c0e49905b0d352c32089b65260c9a3465c9e4408482e"
Sep 09 21:06:17.035416 ip-10-0-138-224 kubenswrapper[2579]: E0909 21:06:17.035398    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fbc6b7d7f58e14e84766c0e49905b0d352c32089b65260c9a3465c9e4408482e\": container with ID starting with fbc6b7d7f58e14e84766c0e49905b0d352c32089b65260c9a3465c9e4408482e not found: ID does not exist" containerID="fbc6b7d7f58e14e84766c0e49905b0d352c32089b65260c9a3465c9e4408482e"
Sep 09 21:06:17.035500 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:06:17.035423    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fbc6b7d7f58e14e84766c0e49905b0d352c32089b65260c9a3465c9e4408482e"} err="failed to get container status \"fbc6b7d7f58e14e84766c0e49905b0d352c32089b65260c9a3465c9e4408482e\": rpc error: code = NotFound desc = could not find container \"fbc6b7d7f58e14e84766c0e49905b0d352c32089b65260c9a3465c9e4408482e\": container with ID starting with fbc6b7d7f58e14e84766c0e49905b0d352c32089b65260c9a3465c9e4408482e not found: ID does not exist"
Sep 09 21:06:17.036282 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:06:17.036262    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-vvwdp"]
Sep 09 21:06:17.046665 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:06:17.046645    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-vvwdp"]
Sep 09 21:06:17.290288 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:06:17.290266    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="340c339e-9ed0-4ba2-b6c0-dad02ef88836" path="/var/lib/kubelet/pods/340c339e-9ed0-4ba2-b6c0-dad02ef88836/volumes"
Sep 09 21:06:17.989115 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:06:17.989082    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5bb8f495f4-ds7lm"]
Sep 09 21:06:17.989477 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:06:17.989403    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c9e563c3-c2c7-44c1-8061-f7730429b4ee" containerName="kserve-container"
Sep 09 21:06:17.989477 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:06:17.989413    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="c9e563c3-c2c7-44c1-8061-f7730429b4ee" containerName="kserve-container"
Sep 09 21:06:17.989477 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:06:17.989421    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c9e563c3-c2c7-44c1-8061-f7730429b4ee" containerName="agent"
Sep 09 21:06:17.989477 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:06:17.989426    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="c9e563c3-c2c7-44c1-8061-f7730429b4ee" containerName="agent"
Sep 09 21:06:17.989477 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:06:17.989450    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c9e563c3-c2c7-44c1-8061-f7730429b4ee" containerName="kube-rbac-proxy"
Sep 09 21:06:17.989477 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:06:17.989456    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="c9e563c3-c2c7-44c1-8061-f7730429b4ee" containerName="kube-rbac-proxy"
Sep 09 21:06:17.989477 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:06:17.989470    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="340c339e-9ed0-4ba2-b6c0-dad02ef88836" containerName="kube-rbac-proxy"
Sep 09 21:06:17.989477 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:06:17.989475    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="340c339e-9ed0-4ba2-b6c0-dad02ef88836" containerName="kube-rbac-proxy"
Sep 09 21:06:17.989477 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:06:17.989481    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="340c339e-9ed0-4ba2-b6c0-dad02ef88836" containerName="storage-initializer"
Sep 09 21:06:17.989743 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:06:17.989486    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="340c339e-9ed0-4ba2-b6c0-dad02ef88836" containerName="storage-initializer"
Sep 09 21:06:17.989743 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:06:17.989497    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c9e563c3-c2c7-44c1-8061-f7730429b4ee" containerName="storage-initializer"
Sep 09 21:06:17.989743 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:06:17.989502    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="c9e563c3-c2c7-44c1-8061-f7730429b4ee" containerName="storage-initializer"
Sep 09 21:06:17.989743 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:06:17.989507    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="340c339e-9ed0-4ba2-b6c0-dad02ef88836" containerName="kserve-container"
Sep 09 21:06:17.989743 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:06:17.989513    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="340c339e-9ed0-4ba2-b6c0-dad02ef88836" containerName="kserve-container"
Sep 09 21:06:17.989743 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:06:17.989559    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="340c339e-9ed0-4ba2-b6c0-dad02ef88836" containerName="kube-rbac-proxy"
Sep 09 21:06:17.989743 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:06:17.989566    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="340c339e-9ed0-4ba2-b6c0-dad02ef88836" containerName="kserve-container"
Sep 09 21:06:17.989743 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:06:17.989573    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="c9e563c3-c2c7-44c1-8061-f7730429b4ee" containerName="agent"
Sep 09 21:06:17.989743 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:06:17.989578    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="c9e563c3-c2c7-44c1-8061-f7730429b4ee" containerName="kserve-container"
Sep 09 21:06:17.989743 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:06:17.989583    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="c9e563c3-c2c7-44c1-8061-f7730429b4ee" containerName="kube-rbac-proxy"
Sep 09 21:06:17.992683 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:06:17.992666    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5bb8f495f4-ds7lm"
Sep 09 21:06:17.995282 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:06:17.995262    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 09 21:06:17.995415 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:06:17.995393    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 09 21:06:17.995677 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:06:17.995658    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-v2-predictor-serving-cert\""
Sep 09 21:06:17.995827 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:06:17.995811    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 09 21:06:17.995877 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:06:17.995855    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 09 21:06:17.996463 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:06:17.996425    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-tlmnz\""
Sep 09 21:06:17.996665 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:06:17.996650    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-kube-rbac-proxy-sar-config\""
Sep 09 21:06:18.007409 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:06:18.007392    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-8xkkd"]
Sep 09 21:06:18.010848 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:06:18.010821    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5bb8f495f4-ds7lm"]
Sep 09 21:06:18.010953 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:06:18.010937    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-8xkkd"
Sep 09 21:06:18.013624 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:06:18.013596    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-predictor-serving-cert\""
Sep 09 21:06:18.026652 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:06:18.026633    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-8xkkd"]
Sep 09 21:06:18.036559 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:06:18.036534    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hc5p\" (UniqueName: \"kubernetes.io/projected/fcf465c7-1a79-47bd-9490-1dc8f09a5fca-kube-api-access-4hc5p\") pod \"isvc-canary-stop-v2-predictor-5bb8f495f4-ds7lm\" (UID: \"fcf465c7-1a79-47bd-9490-1dc8f09a5fca\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5bb8f495f4-ds7lm"
Sep 09 21:06:18.036711 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:06:18.036590    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fcf465c7-1a79-47bd-9490-1dc8f09a5fca-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-5bb8f495f4-ds7lm\" (UID: \"fcf465c7-1a79-47bd-9490-1dc8f09a5fca\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5bb8f495f4-ds7lm"
Sep 09 21:06:18.036711 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:06:18.036623    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fcf465c7-1a79-47bd-9490-1dc8f09a5fca-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-5bb8f495f4-ds7lm\" (UID: \"fcf465c7-1a79-47bd-9490-1dc8f09a5fca\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5bb8f495f4-ds7lm"
Sep 09 21:06:18.036711 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:06:18.036686    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fcf465c7-1a79-47bd-9490-1dc8f09a5fca-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-5bb8f495f4-ds7lm\" (UID: \"fcf465c7-1a79-47bd-9490-1dc8f09a5fca\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5bb8f495f4-ds7lm"
Sep 09 21:06:18.036863 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:06:18.036735    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fcf465c7-1a79-47bd-9490-1dc8f09a5fca-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-5bb8f495f4-ds7lm\" (UID: \"fcf465c7-1a79-47bd-9490-1dc8f09a5fca\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5bb8f495f4-ds7lm"
Sep 09 21:06:18.138010 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:06:18.137985    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/26cfda75-30a0-41d6-87fe-f8466034a4d2-kserve-provision-location\") pod \"isvc-canary-stop-predictor-59fbfd5bc-8xkkd\" (UID: \"26cfda75-30a0-41d6-87fe-f8466034a4d2\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-8xkkd"
Sep 09 21:06:18.138108 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:06:18.138025    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fcf465c7-1a79-47bd-9490-1dc8f09a5fca-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-5bb8f495f4-ds7lm\" (UID: \"fcf465c7-1a79-47bd-9490-1dc8f09a5fca\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5bb8f495f4-ds7lm"
Sep 09 21:06:18.138108 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:06:18.138044    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/26cfda75-30a0-41d6-87fe-f8466034a4d2-proxy-tls\") pod \"isvc-canary-stop-predictor-59fbfd5bc-8xkkd\" (UID: \"26cfda75-30a0-41d6-87fe-f8466034a4d2\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-8xkkd"
Sep 09 21:06:18.138108 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:06:18.138082    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5pqzh\" (UniqueName: \"kubernetes.io/projected/26cfda75-30a0-41d6-87fe-f8466034a4d2-kube-api-access-5pqzh\") pod \"isvc-canary-stop-predictor-59fbfd5bc-8xkkd\" (UID: \"26cfda75-30a0-41d6-87fe-f8466034a4d2\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-8xkkd"
Sep 09 21:06:18.138259 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:06:18.138131    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fcf465c7-1a79-47bd-9490-1dc8f09a5fca-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-5bb8f495f4-ds7lm\" (UID: \"fcf465c7-1a79-47bd-9490-1dc8f09a5fca\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5bb8f495f4-ds7lm"
Sep 09 21:06:18.138259 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:06:18.138169    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hc5p\" (UniqueName: \"kubernetes.io/projected/fcf465c7-1a79-47bd-9490-1dc8f09a5fca-kube-api-access-4hc5p\") pod \"isvc-canary-stop-v2-predictor-5bb8f495f4-ds7lm\" (UID: \"fcf465c7-1a79-47bd-9490-1dc8f09a5fca\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5bb8f495f4-ds7lm"
Sep 09 21:06:18.138259 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:06:18.138189    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/26cfda75-30a0-41d6-87fe-f8466034a4d2-cabundle-cert\") pod \"isvc-canary-stop-predictor-59fbfd5bc-8xkkd\" (UID: \"26cfda75-30a0-41d6-87fe-f8466034a4d2\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-8xkkd"
Sep 09 21:06:18.138259 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:06:18.138217    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fcf465c7-1a79-47bd-9490-1dc8f09a5fca-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-5bb8f495f4-ds7lm\" (UID: \"fcf465c7-1a79-47bd-9490-1dc8f09a5fca\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5bb8f495f4-ds7lm"
Sep 09 21:06:18.138259 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:06:18.138233    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/26cfda75-30a0-41d6-87fe-f8466034a4d2-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-59fbfd5bc-8xkkd\" (UID: \"26cfda75-30a0-41d6-87fe-f8466034a4d2\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-8xkkd"
Sep 09 21:06:18.138582 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:06:18.138273    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fcf465c7-1a79-47bd-9490-1dc8f09a5fca-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-5bb8f495f4-ds7lm\" (UID: \"fcf465c7-1a79-47bd-9490-1dc8f09a5fca\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5bb8f495f4-ds7lm"
Sep 09 21:06:18.138582 ip-10-0-138-224 kubenswrapper[2579]: E0909 21:06:18.138323    2579 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-serving-cert: secret "isvc-canary-stop-v2-predictor-serving-cert" not found
Sep 09 21:06:18.138582 ip-10-0-138-224 kubenswrapper[2579]: E0909 21:06:18.138390    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fcf465c7-1a79-47bd-9490-1dc8f09a5fca-proxy-tls podName:fcf465c7-1a79-47bd-9490-1dc8f09a5fca nodeName:}" failed. No retries permitted until 2026-09-09 21:06:18.638371242 +0000 UTC m=+835.878465343 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/fcf465c7-1a79-47bd-9490-1dc8f09a5fca-proxy-tls") pod "isvc-canary-stop-v2-predictor-5bb8f495f4-ds7lm" (UID: "fcf465c7-1a79-47bd-9490-1dc8f09a5fca") : secret "isvc-canary-stop-v2-predictor-serving-cert" not found
Sep 09 21:06:18.138582 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:06:18.138566    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fcf465c7-1a79-47bd-9490-1dc8f09a5fca-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-5bb8f495f4-ds7lm\" (UID: \"fcf465c7-1a79-47bd-9490-1dc8f09a5fca\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5bb8f495f4-ds7lm"
Sep 09 21:06:18.138821 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:06:18.138674    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fcf465c7-1a79-47bd-9490-1dc8f09a5fca-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-5bb8f495f4-ds7lm\" (UID: \"fcf465c7-1a79-47bd-9490-1dc8f09a5fca\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5bb8f495f4-ds7lm"
Sep 09 21:06:18.138880 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:06:18.138856    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fcf465c7-1a79-47bd-9490-1dc8f09a5fca-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-5bb8f495f4-ds7lm\" (UID: \"fcf465c7-1a79-47bd-9490-1dc8f09a5fca\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5bb8f495f4-ds7lm"
Sep 09 21:06:18.148961 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:06:18.148943    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hc5p\" (UniqueName: \"kubernetes.io/projected/fcf465c7-1a79-47bd-9490-1dc8f09a5fca-kube-api-access-4hc5p\") pod \"isvc-canary-stop-v2-predictor-5bb8f495f4-ds7lm\" (UID: \"fcf465c7-1a79-47bd-9490-1dc8f09a5fca\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5bb8f495f4-ds7lm"
Sep 09 21:06:18.239105 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:06:18.239083    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/26cfda75-30a0-41d6-87fe-f8466034a4d2-cabundle-cert\") pod \"isvc-canary-stop-predictor-59fbfd5bc-8xkkd\" (UID: \"26cfda75-30a0-41d6-87fe-f8466034a4d2\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-8xkkd"
Sep 09 21:06:18.239229 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:06:18.239126    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/26cfda75-30a0-41d6-87fe-f8466034a4d2-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-59fbfd5bc-8xkkd\" (UID: \"26cfda75-30a0-41d6-87fe-f8466034a4d2\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-8xkkd"
Sep 09 21:06:18.239229 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:06:18.239152    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/26cfda75-30a0-41d6-87fe-f8466034a4d2-kserve-provision-location\") pod \"isvc-canary-stop-predictor-59fbfd5bc-8xkkd\" (UID: \"26cfda75-30a0-41d6-87fe-f8466034a4d2\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-8xkkd"
Sep 09 21:06:18.239229 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:06:18.239178    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/26cfda75-30a0-41d6-87fe-f8466034a4d2-proxy-tls\") pod \"isvc-canary-stop-predictor-59fbfd5bc-8xkkd\" (UID: \"26cfda75-30a0-41d6-87fe-f8466034a4d2\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-8xkkd"
Sep 09 21:06:18.239229 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:06:18.239199    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5pqzh\" (UniqueName: \"kubernetes.io/projected/26cfda75-30a0-41d6-87fe-f8466034a4d2-kube-api-access-5pqzh\") pod \"isvc-canary-stop-predictor-59fbfd5bc-8xkkd\" (UID: \"26cfda75-30a0-41d6-87fe-f8466034a4d2\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-8xkkd"
Sep 09 21:06:18.239555 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:06:18.239536    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/26cfda75-30a0-41d6-87fe-f8466034a4d2-kserve-provision-location\") pod \"isvc-canary-stop-predictor-59fbfd5bc-8xkkd\" (UID: \"26cfda75-30a0-41d6-87fe-f8466034a4d2\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-8xkkd"
Sep 09 21:06:18.239679 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:06:18.239662    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/26cfda75-30a0-41d6-87fe-f8466034a4d2-cabundle-cert\") pod \"isvc-canary-stop-predictor-59fbfd5bc-8xkkd\" (UID: \"26cfda75-30a0-41d6-87fe-f8466034a4d2\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-8xkkd"
Sep 09 21:06:18.239757 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:06:18.239732    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/26cfda75-30a0-41d6-87fe-f8466034a4d2-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-59fbfd5bc-8xkkd\" (UID: \"26cfda75-30a0-41d6-87fe-f8466034a4d2\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-8xkkd"
Sep 09 21:06:18.241653 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:06:18.241635    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/26cfda75-30a0-41d6-87fe-f8466034a4d2-proxy-tls\") pod \"isvc-canary-stop-predictor-59fbfd5bc-8xkkd\" (UID: \"26cfda75-30a0-41d6-87fe-f8466034a4d2\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-8xkkd"
Sep 09 21:06:18.247580 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:06:18.247559    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5pqzh\" (UniqueName: \"kubernetes.io/projected/26cfda75-30a0-41d6-87fe-f8466034a4d2-kube-api-access-5pqzh\") pod \"isvc-canary-stop-predictor-59fbfd5bc-8xkkd\" (UID: \"26cfda75-30a0-41d6-87fe-f8466034a4d2\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-8xkkd"
Sep 09 21:06:18.321008 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:06:18.320985    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-8xkkd"
Sep 09 21:06:18.605670 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:06:18.605644    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-8xkkd"]
Sep 09 21:06:18.608034 ip-10-0-138-224 kubenswrapper[2579]: W0909 21:06:18.608005    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26cfda75_30a0_41d6_87fe_f8466034a4d2.slice/crio-02b736b2966d0bfc642629a2fdad489eab96f38071689b85da044e506c90886f WatchSource:0}: Error finding container 02b736b2966d0bfc642629a2fdad489eab96f38071689b85da044e506c90886f: Status 404 returned error can't find the container with id 02b736b2966d0bfc642629a2fdad489eab96f38071689b85da044e506c90886f
Sep 09 21:06:18.642522 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:06:18.642495    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fcf465c7-1a79-47bd-9490-1dc8f09a5fca-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-5bb8f495f4-ds7lm\" (UID: \"fcf465c7-1a79-47bd-9490-1dc8f09a5fca\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5bb8f495f4-ds7lm"
Sep 09 21:06:18.645259 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:06:18.645234    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fcf465c7-1a79-47bd-9490-1dc8f09a5fca-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-5bb8f495f4-ds7lm\" (UID: \"fcf465c7-1a79-47bd-9490-1dc8f09a5fca\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5bb8f495f4-ds7lm"
Sep 09 21:06:18.903365 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:06:18.903289    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5bb8f495f4-ds7lm"
Sep 09 21:06:19.020963 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:06:19.020929    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-8xkkd" event={"ID":"26cfda75-30a0-41d6-87fe-f8466034a4d2","Type":"ContainerStarted","Data":"519ea8fb7c5254dc00538850b5d29d1ea985bb412f1583cc9f8f6dd641a205fc"}
Sep 09 21:06:19.021310 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:06:19.021003    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-8xkkd" event={"ID":"26cfda75-30a0-41d6-87fe-f8466034a4d2","Type":"ContainerStarted","Data":"02b736b2966d0bfc642629a2fdad489eab96f38071689b85da044e506c90886f"}
Sep 09 21:06:19.178131 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:06:19.178105    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5bb8f495f4-ds7lm"]
Sep 09 21:06:19.180431 ip-10-0-138-224 kubenswrapper[2579]: W0909 21:06:19.180400    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcf465c7_1a79_47bd_9490_1dc8f09a5fca.slice/crio-e424b7744415382524a14f5cf162455f154e166561438f085b43e9c4990cab61 WatchSource:0}: Error finding container e424b7744415382524a14f5cf162455f154e166561438f085b43e9c4990cab61: Status 404 returned error can't find the container with id e424b7744415382524a14f5cf162455f154e166561438f085b43e9c4990cab61
Sep 09 21:06:20.027021 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:06:20.026983    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5bb8f495f4-ds7lm" event={"ID":"fcf465c7-1a79-47bd-9490-1dc8f09a5fca","Type":"ContainerStarted","Data":"ab352443dd91393f9f9e996cf4933a784f6b659ba92127759ba70432b2b15bd0"}
Sep 09 21:06:20.027502 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:06:20.027042    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5bb8f495f4-ds7lm" event={"ID":"fcf465c7-1a79-47bd-9490-1dc8f09a5fca","Type":"ContainerStarted","Data":"e424b7744415382524a14f5cf162455f154e166561438f085b43e9c4990cab61"}
Sep 09 21:06:23.040920 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:06:23.040874    2579 generic.go:356] "Generic (PLEG): container finished" podID="26cfda75-30a0-41d6-87fe-f8466034a4d2" containerID="519ea8fb7c5254dc00538850b5d29d1ea985bb412f1583cc9f8f6dd641a205fc" exitCode=0
Sep 09 21:06:23.041291 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:06:23.040943    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-8xkkd" event={"ID":"26cfda75-30a0-41d6-87fe-f8466034a4d2","Type":"ContainerDied","Data":"519ea8fb7c5254dc00538850b5d29d1ea985bb412f1583cc9f8f6dd641a205fc"}
Sep 09 21:06:24.047544 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:06:24.047506    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-8xkkd" event={"ID":"26cfda75-30a0-41d6-87fe-f8466034a4d2","Type":"ContainerStarted","Data":"6a71ee3731a95960b29812b827e35862a2415798a6a963ab5d71541a28b49d62"}
Sep 09 21:06:24.047950 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:06:24.047571    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-8xkkd" event={"ID":"26cfda75-30a0-41d6-87fe-f8466034a4d2","Type":"ContainerStarted","Data":"eceebf8df5fead337a32f1bdb0aa0f73c1693172e8835f433d579f3881c41868"}
Sep 09 21:06:24.048012 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:06:24.047935    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-8xkkd"
Sep 09 21:06:24.048100 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:06:24.048084    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-8xkkd"
Sep 09 21:06:24.049171 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:06:24.049149    2579 generic.go:356] "Generic (PLEG): container finished" podID="fcf465c7-1a79-47bd-9490-1dc8f09a5fca" containerID="ab352443dd91393f9f9e996cf4933a784f6b659ba92127759ba70432b2b15bd0" exitCode=0
Sep 09 21:06:24.049275 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:06:24.049183    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5bb8f495f4-ds7lm" event={"ID":"fcf465c7-1a79-47bd-9490-1dc8f09a5fca","Type":"ContainerDied","Data":"ab352443dd91393f9f9e996cf4933a784f6b659ba92127759ba70432b2b15bd0"}
Sep 09 21:06:24.049681 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:06:24.049659    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-8xkkd" podUID="26cfda75-30a0-41d6-87fe-f8466034a4d2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 09 21:06:24.068050 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:06:24.068000    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-8xkkd" podStartSLOduration=3.179130716 podStartE2EDuration="7.067981248s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.888850532s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:06:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:06:24.065952235 +0000 UTC m=+841.306046390" watchObservedRunningTime="2026-09-09 21:06:24.067981248 +0000 UTC m=+841.308075368"
Sep 09 21:06:25.056169 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:06:25.056134    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5bb8f495f4-ds7lm" event={"ID":"fcf465c7-1a79-47bd-9490-1dc8f09a5fca","Type":"ContainerStarted","Data":"0dbc840ec11c33b39f98eb25dbdc59475746b5f300ca921a188d61c9bcfd25ae"}
Sep 09 21:06:25.056644 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:06:25.056234    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5bb8f495f4-ds7lm" event={"ID":"fcf465c7-1a79-47bd-9490-1dc8f09a5fca","Type":"ContainerStarted","Data":"a7d51b59e97a8fee5c7b22eadb7a27b36658d9b916c5dabc667304c2532d1b06"}
Sep 09 21:06:25.056644 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:06:25.056393    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-8xkkd" podUID="26cfda75-30a0-41d6-87fe-f8466034a4d2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 09 21:06:25.056644 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:06:25.056605    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5bb8f495f4-ds7lm"
Sep 09 21:06:25.056862 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:06:25.056750    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5bb8f495f4-ds7lm"
Sep 09 21:06:25.057574 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:06:25.057554    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5bb8f495f4-ds7lm" podUID="fcf465c7-1a79-47bd-9490-1dc8f09a5fca" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 09 21:06:25.079055 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:06:25.079001    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5bb8f495f4-ds7lm" podStartSLOduration=4.27537622 podStartE2EDuration="8.078988901s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.803612681s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:06:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:06:25.076536319 +0000 UTC m=+842.316630437" watchObservedRunningTime="2026-09-09 21:06:25.078988901 +0000 UTC m=+842.319083020"
Sep 09 21:06:26.060599 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:06:26.060565    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5bb8f495f4-ds7lm" podUID="fcf465c7-1a79-47bd-9490-1dc8f09a5fca" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 09 21:06:27.891265 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:06:27.891231    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-c959c-6754b6cb7-k6mdr"]
Sep 09 21:06:27.894664 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:06:27.894647    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-raw-c959c-6754b6cb7-k6mdr"
Sep 09 21:06:27.897097 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:06:27.897073    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"model-chainer-raw-c959c-serving-cert\""
Sep 09 21:06:27.897203 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:06:27.897073    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"model-chainer-raw-c959c-kube-rbac-proxy-sar-config\""
Sep 09 21:06:27.903044 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:06:27.903019    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-c959c-6754b6cb7-k6mdr"]
Sep 09 21:06:28.021086 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:06:28.021053    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0a25dae2-a07b-4d1b-a439-15b2bd1b2915-openshift-service-ca-bundle\") pod \"model-chainer-raw-c959c-6754b6cb7-k6mdr\" (UID: \"0a25dae2-a07b-4d1b-a439-15b2bd1b2915\") " pod="kserve-ci-e2e-test/model-chainer-raw-c959c-6754b6cb7-k6mdr"
Sep 09 21:06:28.021224 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:06:28.021093    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0a25dae2-a07b-4d1b-a439-15b2bd1b2915-proxy-tls\") pod \"model-chainer-raw-c959c-6754b6cb7-k6mdr\" (UID: \"0a25dae2-a07b-4d1b-a439-15b2bd1b2915\") " pod="kserve-ci-e2e-test/model-chainer-raw-c959c-6754b6cb7-k6mdr"
Sep 09 21:06:28.121931 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:06:28.121900    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0a25dae2-a07b-4d1b-a439-15b2bd1b2915-proxy-tls\") pod \"model-chainer-raw-c959c-6754b6cb7-k6mdr\" (UID: \"0a25dae2-a07b-4d1b-a439-15b2bd1b2915\") " pod="kserve-ci-e2e-test/model-chainer-raw-c959c-6754b6cb7-k6mdr"
Sep 09 21:06:28.122129 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:06:28.122002    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0a25dae2-a07b-4d1b-a439-15b2bd1b2915-openshift-service-ca-bundle\") pod \"model-chainer-raw-c959c-6754b6cb7-k6mdr\" (UID: \"0a25dae2-a07b-4d1b-a439-15b2bd1b2915\") " pod="kserve-ci-e2e-test/model-chainer-raw-c959c-6754b6cb7-k6mdr"
Sep 09 21:06:28.122129 ip-10-0-138-224 kubenswrapper[2579]: E0909 21:06:28.122071    2579 secret.go:189] Couldn't get secret kserve-ci-e2e-test/model-chainer-raw-c959c-serving-cert: secret "model-chainer-raw-c959c-serving-cert" not found
Sep 09 21:06:28.122259 ip-10-0-138-224 kubenswrapper[2579]: E0909 21:06:28.122147    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a25dae2-a07b-4d1b-a439-15b2bd1b2915-proxy-tls podName:0a25dae2-a07b-4d1b-a439-15b2bd1b2915 nodeName:}" failed. No retries permitted until 2026-09-09 21:06:28.62212653 +0000 UTC m=+845.862220631 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/0a25dae2-a07b-4d1b-a439-15b2bd1b2915-proxy-tls") pod "model-chainer-raw-c959c-6754b6cb7-k6mdr" (UID: "0a25dae2-a07b-4d1b-a439-15b2bd1b2915") : secret "model-chainer-raw-c959c-serving-cert" not found
Sep 09 21:06:28.122588 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:06:28.122571    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0a25dae2-a07b-4d1b-a439-15b2bd1b2915-openshift-service-ca-bundle\") pod \"model-chainer-raw-c959c-6754b6cb7-k6mdr\" (UID: \"0a25dae2-a07b-4d1b-a439-15b2bd1b2915\") " pod="kserve-ci-e2e-test/model-chainer-raw-c959c-6754b6cb7-k6mdr"
Sep 09 21:06:28.626636 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:06:28.626604    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0a25dae2-a07b-4d1b-a439-15b2bd1b2915-proxy-tls\") pod \"model-chainer-raw-c959c-6754b6cb7-k6mdr\" (UID: \"0a25dae2-a07b-4d1b-a439-15b2bd1b2915\") " pod="kserve-ci-e2e-test/model-chainer-raw-c959c-6754b6cb7-k6mdr"
Sep 09 21:06:28.629102 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:06:28.629074    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0a25dae2-a07b-4d1b-a439-15b2bd1b2915-proxy-tls\") pod \"model-chainer-raw-c959c-6754b6cb7-k6mdr\" (UID: \"0a25dae2-a07b-4d1b-a439-15b2bd1b2915\") " pod="kserve-ci-e2e-test/model-chainer-raw-c959c-6754b6cb7-k6mdr"
Sep 09 21:06:28.806143 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:06:28.806110    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-raw-c959c-6754b6cb7-k6mdr"
Sep 09 21:06:29.075741 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:06:29.075713    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-c959c-6754b6cb7-k6mdr"]
Sep 09 21:06:29.077732 ip-10-0-138-224 kubenswrapper[2579]: W0909 21:06:29.077704    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a25dae2_a07b_4d1b_a439_15b2bd1b2915.slice/crio-9909e330448c44c134ce7fe2652cd15e8d3f157011d81ac19633ad811111d504 WatchSource:0}: Error finding container 9909e330448c44c134ce7fe2652cd15e8d3f157011d81ac19633ad811111d504: Status 404 returned error can't find the container with id 9909e330448c44c134ce7fe2652cd15e8d3f157011d81ac19633ad811111d504
Sep 09 21:06:30.063016 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:06:30.062980    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-8xkkd"
Sep 09 21:06:30.064427 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:06:30.063878    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-8xkkd" podUID="26cfda75-30a0-41d6-87fe-f8466034a4d2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 09 21:06:30.084116 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:06:30.084078    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-raw-c959c-6754b6cb7-k6mdr" event={"ID":"0a25dae2-a07b-4d1b-a439-15b2bd1b2915","Type":"ContainerStarted","Data":"9909e330448c44c134ce7fe2652cd15e8d3f157011d81ac19633ad811111d504"}
Sep 09 21:06:31.066557 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:06:31.066528    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5bb8f495f4-ds7lm"
Sep 09 21:06:31.067151 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:06:31.067120    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5bb8f495f4-ds7lm" podUID="fcf465c7-1a79-47bd-9490-1dc8f09a5fca" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 09 21:06:32.099493 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:06:32.099453    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-raw-c959c-6754b6cb7-k6mdr" event={"ID":"0a25dae2-a07b-4d1b-a439-15b2bd1b2915","Type":"ContainerStarted","Data":"101dc79cd8c1da4fe3b7471b59dd22e4edaccc540438ecdaba0da14f0ad31c39"}
Sep 09 21:06:32.099915 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:06:32.099572    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/model-chainer-raw-c959c-6754b6cb7-k6mdr"
Sep 09 21:06:32.115126 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:06:32.115060    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/model-chainer-raw-c959c-6754b6cb7-k6mdr" podStartSLOduration=2.815103518 podStartE2EDuration="5.115046033s" totalImagesPullingTime="2.299942515s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:06:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:06:32.114326553 +0000 UTC m=+849.354420671" watchObservedRunningTime="2026-09-09 21:06:32.115046033 +0000 UTC m=+849.355140197"
Sep 09 21:06:37.940072 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:06:37.940032    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-c959c-6754b6cb7-k6mdr"]
Sep 09 21:06:37.940504 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:06:37.940201    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/model-chainer-raw-c959c-6754b6cb7-k6mdr" podUID="0a25dae2-a07b-4d1b-a439-15b2bd1b2915" containerName="model-chainer-raw-c959c" containerID="cri-o://101dc79cd8c1da4fe3b7471b59dd22e4edaccc540438ecdaba0da14f0ad31c39" gracePeriod=30
Sep 09 21:06:37.945283 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:06:37.945258    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-c959c-6754b6cb7-k6mdr" podUID="0a25dae2-a07b-4d1b-a439-15b2bd1b2915" containerName="model-chainer-raw-c959c" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 21:06:38.213134 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:06:38.213091    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-887b3-predictor-5ddcd6d77d-gs5gp"]
Sep 09 21:06:38.216931 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:06:38.216907    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-887b3-predictor-5ddcd6d77d-gs5gp"
Sep 09 21:06:38.219783 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:06:38.219747    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-hpa-887b3-predictor-serving-cert\""
Sep 09 21:06:38.219944 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:06:38.219926    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-hpa-887b3-kube-rbac-proxy-sar-config\""
Sep 09 21:06:38.238123 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:06:38.238098    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-887b3-predictor-5ddcd6d77d-gs5gp"]
Sep 09 21:06:38.314997 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:06:38.314964    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c0ff6dd4-d542-4467-8222-702149441465-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-887b3-predictor-5ddcd6d77d-gs5gp\" (UID: \"c0ff6dd4-d542-4467-8222-702149441465\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-887b3-predictor-5ddcd6d77d-gs5gp"
Sep 09 21:06:38.315152 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:06:38.315115    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c0ff6dd4-d542-4467-8222-702149441465-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-887b3-predictor-5ddcd6d77d-gs5gp\" (UID: \"c0ff6dd4-d542-4467-8222-702149441465\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-887b3-predictor-5ddcd6d77d-gs5gp"
Sep 09 21:06:38.315202 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:06:38.315163    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c0ff6dd4-d542-4467-8222-702149441465-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-887b3-predictor-5ddcd6d77d-gs5gp\" (UID: \"c0ff6dd4-d542-4467-8222-702149441465\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-887b3-predictor-5ddcd6d77d-gs5gp"
Sep 09 21:06:38.315274 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:06:38.315254    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzblt\" (UniqueName: \"kubernetes.io/projected/c0ff6dd4-d542-4467-8222-702149441465-kube-api-access-bzblt\") pod \"isvc-sklearn-graph-raw-hpa-887b3-predictor-5ddcd6d77d-gs5gp\" (UID: \"c0ff6dd4-d542-4467-8222-702149441465\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-887b3-predictor-5ddcd6d77d-gs5gp"
Sep 09 21:06:38.315307 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:06:38.315294    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c0ff6dd4-d542-4467-8222-702149441465-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-887b3-predictor-5ddcd6d77d-gs5gp\" (UID: \"c0ff6dd4-d542-4467-8222-702149441465\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-887b3-predictor-5ddcd6d77d-gs5gp"
Sep 09 21:06:38.416753 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:06:38.416718    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzblt\" (UniqueName: \"kubernetes.io/projected/c0ff6dd4-d542-4467-8222-702149441465-kube-api-access-bzblt\") pod \"isvc-sklearn-graph-raw-hpa-887b3-predictor-5ddcd6d77d-gs5gp\" (UID: \"c0ff6dd4-d542-4467-8222-702149441465\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-887b3-predictor-5ddcd6d77d-gs5gp"
Sep 09 21:06:38.416919 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:06:38.416758    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c0ff6dd4-d542-4467-8222-702149441465-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-887b3-predictor-5ddcd6d77d-gs5gp\" (UID: \"c0ff6dd4-d542-4467-8222-702149441465\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-887b3-predictor-5ddcd6d77d-gs5gp"
Sep 09 21:06:38.416919 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:06:38.416798    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c0ff6dd4-d542-4467-8222-702149441465-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-887b3-predictor-5ddcd6d77d-gs5gp\" (UID: \"c0ff6dd4-d542-4467-8222-702149441465\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-887b3-predictor-5ddcd6d77d-gs5gp"
Sep 09 21:06:38.416919 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:06:38.416872    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c0ff6dd4-d542-4467-8222-702149441465-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-887b3-predictor-5ddcd6d77d-gs5gp\" (UID: \"c0ff6dd4-d542-4467-8222-702149441465\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-887b3-predictor-5ddcd6d77d-gs5gp"
Sep 09 21:06:38.416919 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:06:38.416907    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c0ff6dd4-d542-4467-8222-702149441465-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-887b3-predictor-5ddcd6d77d-gs5gp\" (UID: \"c0ff6dd4-d542-4467-8222-702149441465\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-887b3-predictor-5ddcd6d77d-gs5gp"
Sep 09 21:06:38.417350 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:06:38.417326    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c0ff6dd4-d542-4467-8222-702149441465-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-887b3-predictor-5ddcd6d77d-gs5gp\" (UID: \"c0ff6dd4-d542-4467-8222-702149441465\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-887b3-predictor-5ddcd6d77d-gs5gp"
Sep 09 21:06:38.417571 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:06:38.417548    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c0ff6dd4-d542-4467-8222-702149441465-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-887b3-predictor-5ddcd6d77d-gs5gp\" (UID: \"c0ff6dd4-d542-4467-8222-702149441465\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-887b3-predictor-5ddcd6d77d-gs5gp"
Sep 09 21:06:38.417660 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:06:38.417581    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c0ff6dd4-d542-4467-8222-702149441465-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-887b3-predictor-5ddcd6d77d-gs5gp\" (UID: \"c0ff6dd4-d542-4467-8222-702149441465\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-887b3-predictor-5ddcd6d77d-gs5gp"
Sep 09 21:06:38.419659 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:06:38.419638    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c0ff6dd4-d542-4467-8222-702149441465-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-887b3-predictor-5ddcd6d77d-gs5gp\" (UID: \"c0ff6dd4-d542-4467-8222-702149441465\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-887b3-predictor-5ddcd6d77d-gs5gp"
Sep 09 21:06:38.428178 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:06:38.428150    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzblt\" (UniqueName: \"kubernetes.io/projected/c0ff6dd4-d542-4467-8222-702149441465-kube-api-access-bzblt\") pod \"isvc-sklearn-graph-raw-hpa-887b3-predictor-5ddcd6d77d-gs5gp\" (UID: \"c0ff6dd4-d542-4467-8222-702149441465\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-887b3-predictor-5ddcd6d77d-gs5gp"
Sep 09 21:06:38.528242 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:06:38.528171    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-887b3-predictor-5ddcd6d77d-gs5gp"
Sep 09 21:06:38.809527 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:06:38.809502    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-887b3-predictor-5ddcd6d77d-gs5gp"]
Sep 09 21:06:38.811328 ip-10-0-138-224 kubenswrapper[2579]: W0909 21:06:38.811296    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0ff6dd4_d542_4467_8222_702149441465.slice/crio-1daae64f2c02a2be2609e54e3f93ba268fa75d4174fc71d2676080d01421859a WatchSource:0}: Error finding container 1daae64f2c02a2be2609e54e3f93ba268fa75d4174fc71d2676080d01421859a: Status 404 returned error can't find the container with id 1daae64f2c02a2be2609e54e3f93ba268fa75d4174fc71d2676080d01421859a
Sep 09 21:06:39.133124 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:06:39.133042    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-887b3-predictor-5ddcd6d77d-gs5gp" event={"ID":"c0ff6dd4-d542-4467-8222-702149441465","Type":"ContainerStarted","Data":"6ed37560c5c70358baa1b4db9b6400db61a823b8f088c2c4bb6fade5f2999b22"}
Sep 09 21:06:39.133124 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:06:39.133099    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-887b3-predictor-5ddcd6d77d-gs5gp" event={"ID":"c0ff6dd4-d542-4467-8222-702149441465","Type":"ContainerStarted","Data":"1daae64f2c02a2be2609e54e3f93ba268fa75d4174fc71d2676080d01421859a"}
Sep 09 21:06:40.064133 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:06:40.064092    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-8xkkd" podUID="26cfda75-30a0-41d6-87fe-f8466034a4d2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 09 21:06:41.067519 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:06:41.067479    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5bb8f495f4-ds7lm" podUID="fcf465c7-1a79-47bd-9490-1dc8f09a5fca" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 09 21:06:42.944281 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:06:42.944245    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-c959c-6754b6cb7-k6mdr" podUID="0a25dae2-a07b-4d1b-a439-15b2bd1b2915" containerName="model-chainer-raw-c959c" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 21:06:43.153007 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:06:43.152975    2579 generic.go:356] "Generic (PLEG): container finished" podID="c0ff6dd4-d542-4467-8222-702149441465" containerID="6ed37560c5c70358baa1b4db9b6400db61a823b8f088c2c4bb6fade5f2999b22" exitCode=0
Sep 09 21:06:43.153166 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:06:43.153021    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-887b3-predictor-5ddcd6d77d-gs5gp" event={"ID":"c0ff6dd4-d542-4467-8222-702149441465","Type":"ContainerDied","Data":"6ed37560c5c70358baa1b4db9b6400db61a823b8f088c2c4bb6fade5f2999b22"}
Sep 09 21:06:44.159848 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:06:44.159812    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-887b3-predictor-5ddcd6d77d-gs5gp" event={"ID":"c0ff6dd4-d542-4467-8222-702149441465","Type":"ContainerStarted","Data":"9dd50d2048a6ddea512db058809656e04cecb8470ef6a3c44eaec16fdb5ab2a1"}
Sep 09 21:06:44.160301 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:06:44.159878    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-887b3-predictor-5ddcd6d77d-gs5gp" event={"ID":"c0ff6dd4-d542-4467-8222-702149441465","Type":"ContainerStarted","Data":"4be51b774b82a0473bce1a544c428c3cc718a1aaaef67f927a8f3300ba30e629"}
Sep 09 21:06:44.160301 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:06:44.160087    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-887b3-predictor-5ddcd6d77d-gs5gp"
Sep 09 21:06:44.180717 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:06:44.180650    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-887b3-predictor-5ddcd6d77d-gs5gp" podStartSLOduration=2.612309099 podStartE2EDuration="6.180628063s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.568318964s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:06:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:06:44.17819242 +0000 UTC m=+861.418286552" watchObservedRunningTime="2026-09-09 21:06:44.180628063 +0000 UTC m=+861.420722186"
Sep 09 21:06:45.164381 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:06:45.164349    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-887b3-predictor-5ddcd6d77d-gs5gp"
Sep 09 21:06:45.165599 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:06:45.165571    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-887b3-predictor-5ddcd6d77d-gs5gp" podUID="c0ff6dd4-d542-4467-8222-702149441465" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 09 21:06:46.169508 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:06:46.169463    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-887b3-predictor-5ddcd6d77d-gs5gp" podUID="c0ff6dd4-d542-4467-8222-702149441465" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 09 21:06:47.944986 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:06:47.944932    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-c959c-6754b6cb7-k6mdr" podUID="0a25dae2-a07b-4d1b-a439-15b2bd1b2915" containerName="model-chainer-raw-c959c" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 21:06:50.064002 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:06:50.063966    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-8xkkd" podUID="26cfda75-30a0-41d6-87fe-f8466034a4d2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 09 21:06:51.068039 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:06:51.068000    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5bb8f495f4-ds7lm" podUID="fcf465c7-1a79-47bd-9490-1dc8f09a5fca" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 09 21:06:51.173365 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:06:51.173339    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-887b3-predictor-5ddcd6d77d-gs5gp"
Sep 09 21:06:51.173896 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:06:51.173870    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-887b3-predictor-5ddcd6d77d-gs5gp" podUID="c0ff6dd4-d542-4467-8222-702149441465" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 09 21:06:52.944254 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:06:52.944216    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-c959c-6754b6cb7-k6mdr" podUID="0a25dae2-a07b-4d1b-a439-15b2bd1b2915" containerName="model-chainer-raw-c959c" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 21:06:57.944867 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:06:57.944832    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-c959c-6754b6cb7-k6mdr" podUID="0a25dae2-a07b-4d1b-a439-15b2bd1b2915" containerName="model-chainer-raw-c959c" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 21:07:00.064255 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:07:00.064217    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-8xkkd" podUID="26cfda75-30a0-41d6-87fe-f8466034a4d2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 09 21:07:01.067555 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:07:01.067515    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5bb8f495f4-ds7lm" podUID="fcf465c7-1a79-47bd-9490-1dc8f09a5fca" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 09 21:07:01.173885 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:07:01.173852    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-887b3-predictor-5ddcd6d77d-gs5gp" podUID="c0ff6dd4-d542-4467-8222-702149441465" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 09 21:07:02.944641 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:07:02.944605    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-c959c-6754b6cb7-k6mdr" podUID="0a25dae2-a07b-4d1b-a439-15b2bd1b2915" containerName="model-chainer-raw-c959c" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 21:07:07.945153 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:07:07.945114    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-c959c-6754b6cb7-k6mdr" podUID="0a25dae2-a07b-4d1b-a439-15b2bd1b2915" containerName="model-chainer-raw-c959c" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 21:07:08.100151 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:07:08.100129    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-raw-c959c-6754b6cb7-k6mdr"
Sep 09 21:07:08.176176 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:07:08.176148    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0a25dae2-a07b-4d1b-a439-15b2bd1b2915-openshift-service-ca-bundle\") pod \"0a25dae2-a07b-4d1b-a439-15b2bd1b2915\" (UID: \"0a25dae2-a07b-4d1b-a439-15b2bd1b2915\") "
Sep 09 21:07:08.176318 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:07:08.176257    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0a25dae2-a07b-4d1b-a439-15b2bd1b2915-proxy-tls\") pod \"0a25dae2-a07b-4d1b-a439-15b2bd1b2915\" (UID: \"0a25dae2-a07b-4d1b-a439-15b2bd1b2915\") "
Sep 09 21:07:08.176557 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:07:08.176537    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a25dae2-a07b-4d1b-a439-15b2bd1b2915-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "0a25dae2-a07b-4d1b-a439-15b2bd1b2915" (UID: "0a25dae2-a07b-4d1b-a439-15b2bd1b2915"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:07:08.178434 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:07:08.178409    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a25dae2-a07b-4d1b-a439-15b2bd1b2915-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "0a25dae2-a07b-4d1b-a439-15b2bd1b2915" (UID: "0a25dae2-a07b-4d1b-a439-15b2bd1b2915"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:07:08.271367 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:07:08.271291    2579 generic.go:356] "Generic (PLEG): container finished" podID="0a25dae2-a07b-4d1b-a439-15b2bd1b2915" containerID="101dc79cd8c1da4fe3b7471b59dd22e4edaccc540438ecdaba0da14f0ad31c39" exitCode=137
Sep 09 21:07:08.271367 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:07:08.271334    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-raw-c959c-6754b6cb7-k6mdr"
Sep 09 21:07:08.271590 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:07:08.271385    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-raw-c959c-6754b6cb7-k6mdr" event={"ID":"0a25dae2-a07b-4d1b-a439-15b2bd1b2915","Type":"ContainerDied","Data":"101dc79cd8c1da4fe3b7471b59dd22e4edaccc540438ecdaba0da14f0ad31c39"}
Sep 09 21:07:08.271590 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:07:08.271465    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-raw-c959c-6754b6cb7-k6mdr" event={"ID":"0a25dae2-a07b-4d1b-a439-15b2bd1b2915","Type":"ContainerDied","Data":"9909e330448c44c134ce7fe2652cd15e8d3f157011d81ac19633ad811111d504"}
Sep 09 21:07:08.271590 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:07:08.271495    2579 scope.go:117] "RemoveContainer" containerID="101dc79cd8c1da4fe3b7471b59dd22e4edaccc540438ecdaba0da14f0ad31c39"
Sep 09 21:07:08.276853 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:07:08.276831    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0a25dae2-a07b-4d1b-a439-15b2bd1b2915-openshift-service-ca-bundle\") on node \"ip-10-0-138-224.ec2.internal\" DevicePath \"\""
Sep 09 21:07:08.276946 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:07:08.276860    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0a25dae2-a07b-4d1b-a439-15b2bd1b2915-proxy-tls\") on node \"ip-10-0-138-224.ec2.internal\" DevicePath \"\""
Sep 09 21:07:08.281283 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:07:08.281250    2579 scope.go:117] "RemoveContainer" containerID="101dc79cd8c1da4fe3b7471b59dd22e4edaccc540438ecdaba0da14f0ad31c39"
Sep 09 21:07:08.281584 ip-10-0-138-224 kubenswrapper[2579]: E0909 21:07:08.281556    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"101dc79cd8c1da4fe3b7471b59dd22e4edaccc540438ecdaba0da14f0ad31c39\": container with ID starting with 101dc79cd8c1da4fe3b7471b59dd22e4edaccc540438ecdaba0da14f0ad31c39 not found: ID does not exist" containerID="101dc79cd8c1da4fe3b7471b59dd22e4edaccc540438ecdaba0da14f0ad31c39"
Sep 09 21:07:08.281679 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:07:08.281595    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"101dc79cd8c1da4fe3b7471b59dd22e4edaccc540438ecdaba0da14f0ad31c39"} err="failed to get container status \"101dc79cd8c1da4fe3b7471b59dd22e4edaccc540438ecdaba0da14f0ad31c39\": rpc error: code = NotFound desc = could not find container \"101dc79cd8c1da4fe3b7471b59dd22e4edaccc540438ecdaba0da14f0ad31c39\": container with ID starting with 101dc79cd8c1da4fe3b7471b59dd22e4edaccc540438ecdaba0da14f0ad31c39 not found: ID does not exist"
Sep 09 21:07:08.291790 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:07:08.291763    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-c959c-6754b6cb7-k6mdr"]
Sep 09 21:07:08.294064 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:07:08.294042    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-c959c-6754b6cb7-k6mdr"]
Sep 09 21:07:09.292154 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:07:09.292076    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0a25dae2-a07b-4d1b-a439-15b2bd1b2915" path="/var/lib/kubelet/pods/0a25dae2-a07b-4d1b-a439-15b2bd1b2915/volumes"
Sep 09 21:07:10.064213 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:07:10.064171    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-8xkkd" podUID="26cfda75-30a0-41d6-87fe-f8466034a4d2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 09 21:07:11.067251 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:07:11.067211    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5bb8f495f4-ds7lm" podUID="fcf465c7-1a79-47bd-9490-1dc8f09a5fca" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 09 21:07:11.174638 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:07:11.174596    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-887b3-predictor-5ddcd6d77d-gs5gp" podUID="c0ff6dd4-d542-4467-8222-702149441465" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 09 21:07:20.063804 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:07:20.063716    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-8xkkd" podUID="26cfda75-30a0-41d6-87fe-f8466034a4d2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 09 21:07:21.067670 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:07:21.067632    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5bb8f495f4-ds7lm" podUID="fcf465c7-1a79-47bd-9490-1dc8f09a5fca" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 09 21:07:21.173812 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:07:21.173776    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-887b3-predictor-5ddcd6d77d-gs5gp" podUID="c0ff6dd4-d542-4467-8222-702149441465" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 09 21:07:30.064174 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:07:30.064129    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-8xkkd" podUID="26cfda75-30a0-41d6-87fe-f8466034a4d2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 09 21:07:31.067280 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:07:31.067215    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5bb8f495f4-ds7lm" podUID="fcf465c7-1a79-47bd-9490-1dc8f09a5fca" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 09 21:07:31.174315 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:07:31.174277    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-887b3-predictor-5ddcd6d77d-gs5gp" podUID="c0ff6dd4-d542-4467-8222-702149441465" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 09 21:07:40.064434 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:07:40.064396    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-8xkkd"
Sep 09 21:07:41.068365 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:07:41.068332    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5bb8f495f4-ds7lm"
Sep 09 21:07:41.173834 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:07:41.173792    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-887b3-predictor-5ddcd6d77d-gs5gp" podUID="c0ff6dd4-d542-4467-8222-702149441465" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 09 21:07:47.981933 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:07:47.981898    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5bb8f495f4-ds7lm"]
Sep 09 21:07:47.982415 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:07:47.982167    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5bb8f495f4-ds7lm" podUID="fcf465c7-1a79-47bd-9490-1dc8f09a5fca" containerName="kserve-container" containerID="cri-o://a7d51b59e97a8fee5c7b22eadb7a27b36658d9b916c5dabc667304c2532d1b06" gracePeriod=30
Sep 09 21:07:47.982415 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:07:47.982213    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5bb8f495f4-ds7lm" podUID="fcf465c7-1a79-47bd-9490-1dc8f09a5fca" containerName="kube-rbac-proxy" containerID="cri-o://0dbc840ec11c33b39f98eb25dbdc59475746b5f300ca921a188d61c9bcfd25ae" gracePeriod=30
Sep 09 21:07:48.020470 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:07:48.020420    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-8xkkd"]
Sep 09 21:07:48.020757 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:07:48.020722    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-8xkkd" podUID="26cfda75-30a0-41d6-87fe-f8466034a4d2" containerName="kserve-container" containerID="cri-o://eceebf8df5fead337a32f1bdb0aa0f73c1693172e8835f433d579f3881c41868" gracePeriod=30
Sep 09 21:07:48.020888 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:07:48.020804    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-8xkkd" podUID="26cfda75-30a0-41d6-87fe-f8466034a4d2" containerName="kube-rbac-proxy" containerID="cri-o://6a71ee3731a95960b29812b827e35862a2415798a6a963ab5d71541a28b49d62" gracePeriod=30
Sep 09 21:07:48.469345 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:07:48.469309    2579 generic.go:356] "Generic (PLEG): container finished" podID="26cfda75-30a0-41d6-87fe-f8466034a4d2" containerID="6a71ee3731a95960b29812b827e35862a2415798a6a963ab5d71541a28b49d62" exitCode=2
Sep 09 21:07:48.469545 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:07:48.469383    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-8xkkd" event={"ID":"26cfda75-30a0-41d6-87fe-f8466034a4d2","Type":"ContainerDied","Data":"6a71ee3731a95960b29812b827e35862a2415798a6a963ab5d71541a28b49d62"}
Sep 09 21:07:48.471542 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:07:48.471522    2579 generic.go:356] "Generic (PLEG): container finished" podID="fcf465c7-1a79-47bd-9490-1dc8f09a5fca" containerID="0dbc840ec11c33b39f98eb25dbdc59475746b5f300ca921a188d61c9bcfd25ae" exitCode=2
Sep 09 21:07:48.471630 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:07:48.471571    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5bb8f495f4-ds7lm" event={"ID":"fcf465c7-1a79-47bd-9490-1dc8f09a5fca","Type":"ContainerDied","Data":"0dbc840ec11c33b39f98eb25dbdc59475746b5f300ca921a188d61c9bcfd25ae"}
Sep 09 21:07:50.057431 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:07:50.057388    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-8xkkd" podUID="26cfda75-30a0-41d6-87fe-f8466034a4d2" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.34:8643/healthz\": dial tcp 10.134.0.34:8643: connect: connection refused"
Sep 09 21:07:50.064404 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:07:50.064374    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-8xkkd" podUID="26cfda75-30a0-41d6-87fe-f8466034a4d2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 09 21:07:51.061029 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:07:51.060987    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5bb8f495f4-ds7lm" podUID="fcf465c7-1a79-47bd-9490-1dc8f09a5fca" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.33:8643/healthz\": dial tcp 10.134.0.33:8643: connect: connection refused"
Sep 09 21:07:51.067224 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:07:51.067192    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5bb8f495f4-ds7lm" podUID="fcf465c7-1a79-47bd-9490-1dc8f09a5fca" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 09 21:07:51.174050 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:07:51.174005    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-887b3-predictor-5ddcd6d77d-gs5gp" podUID="c0ff6dd4-d542-4467-8222-702149441465" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 09 21:07:53.487883 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:07:53.487859    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5bb8f495f4-ds7lm"
Sep 09 21:07:53.490638 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:07:53.490617    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-8xkkd"
Sep 09 21:07:53.500338 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:07:53.500311    2579 generic.go:356] "Generic (PLEG): container finished" podID="26cfda75-30a0-41d6-87fe-f8466034a4d2" containerID="eceebf8df5fead337a32f1bdb0aa0f73c1693172e8835f433d579f3881c41868" exitCode=0
Sep 09 21:07:53.500483 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:07:53.500357    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-8xkkd"
Sep 09 21:07:53.500483 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:07:53.500379    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-8xkkd" event={"ID":"26cfda75-30a0-41d6-87fe-f8466034a4d2","Type":"ContainerDied","Data":"eceebf8df5fead337a32f1bdb0aa0f73c1693172e8835f433d579f3881c41868"}
Sep 09 21:07:53.500483 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:07:53.500429    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-8xkkd" event={"ID":"26cfda75-30a0-41d6-87fe-f8466034a4d2","Type":"ContainerDied","Data":"02b736b2966d0bfc642629a2fdad489eab96f38071689b85da044e506c90886f"}
Sep 09 21:07:53.500483 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:07:53.500478    2579 scope.go:117] "RemoveContainer" containerID="6a71ee3731a95960b29812b827e35862a2415798a6a963ab5d71541a28b49d62"
Sep 09 21:07:53.502558 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:07:53.502535    2579 generic.go:356] "Generic (PLEG): container finished" podID="fcf465c7-1a79-47bd-9490-1dc8f09a5fca" containerID="a7d51b59e97a8fee5c7b22eadb7a27b36658d9b916c5dabc667304c2532d1b06" exitCode=0
Sep 09 21:07:53.502661 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:07:53.502572    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5bb8f495f4-ds7lm" event={"ID":"fcf465c7-1a79-47bd-9490-1dc8f09a5fca","Type":"ContainerDied","Data":"a7d51b59e97a8fee5c7b22eadb7a27b36658d9b916c5dabc667304c2532d1b06"}
Sep 09 21:07:53.502661 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:07:53.502576    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5bb8f495f4-ds7lm"
Sep 09 21:07:53.502661 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:07:53.502608    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5bb8f495f4-ds7lm" event={"ID":"fcf465c7-1a79-47bd-9490-1dc8f09a5fca","Type":"ContainerDied","Data":"e424b7744415382524a14f5cf162455f154e166561438f085b43e9c4990cab61"}
Sep 09 21:07:53.509076 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:07:53.509055    2579 scope.go:117] "RemoveContainer" containerID="eceebf8df5fead337a32f1bdb0aa0f73c1693172e8835f433d579f3881c41868"
Sep 09 21:07:53.518083 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:07:53.518066    2579 scope.go:117] "RemoveContainer" containerID="519ea8fb7c5254dc00538850b5d29d1ea985bb412f1583cc9f8f6dd641a205fc"
Sep 09 21:07:53.526140 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:07:53.526120    2579 scope.go:117] "RemoveContainer" containerID="6a71ee3731a95960b29812b827e35862a2415798a6a963ab5d71541a28b49d62"
Sep 09 21:07:53.526429 ip-10-0-138-224 kubenswrapper[2579]: E0909 21:07:53.526412    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a71ee3731a95960b29812b827e35862a2415798a6a963ab5d71541a28b49d62\": container with ID starting with 6a71ee3731a95960b29812b827e35862a2415798a6a963ab5d71541a28b49d62 not found: ID does not exist" containerID="6a71ee3731a95960b29812b827e35862a2415798a6a963ab5d71541a28b49d62"
Sep 09 21:07:53.526510 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:07:53.526453    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a71ee3731a95960b29812b827e35862a2415798a6a963ab5d71541a28b49d62"} err="failed to get container status \"6a71ee3731a95960b29812b827e35862a2415798a6a963ab5d71541a28b49d62\": rpc error: code = NotFound desc = could not find container \"6a71ee3731a95960b29812b827e35862a2415798a6a963ab5d71541a28b49d62\": container with ID starting with 6a71ee3731a95960b29812b827e35862a2415798a6a963ab5d71541a28b49d62 not found: ID does not exist"
Sep 09 21:07:53.526510 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:07:53.526476    2579 scope.go:117] "RemoveContainer" containerID="eceebf8df5fead337a32f1bdb0aa0f73c1693172e8835f433d579f3881c41868"
Sep 09 21:07:53.526742 ip-10-0-138-224 kubenswrapper[2579]: E0909 21:07:53.526723    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eceebf8df5fead337a32f1bdb0aa0f73c1693172e8835f433d579f3881c41868\": container with ID starting with eceebf8df5fead337a32f1bdb0aa0f73c1693172e8835f433d579f3881c41868 not found: ID does not exist" containerID="eceebf8df5fead337a32f1bdb0aa0f73c1693172e8835f433d579f3881c41868"
Sep 09 21:07:53.526816 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:07:53.526750    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eceebf8df5fead337a32f1bdb0aa0f73c1693172e8835f433d579f3881c41868"} err="failed to get container status \"eceebf8df5fead337a32f1bdb0aa0f73c1693172e8835f433d579f3881c41868\": rpc error: code = NotFound desc = could not find container \"eceebf8df5fead337a32f1bdb0aa0f73c1693172e8835f433d579f3881c41868\": container with ID starting with eceebf8df5fead337a32f1bdb0aa0f73c1693172e8835f433d579f3881c41868 not found: ID does not exist"
Sep 09 21:07:53.526816 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:07:53.526772    2579 scope.go:117] "RemoveContainer" containerID="519ea8fb7c5254dc00538850b5d29d1ea985bb412f1583cc9f8f6dd641a205fc"
Sep 09 21:07:53.527063 ip-10-0-138-224 kubenswrapper[2579]: E0909 21:07:53.527047    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"519ea8fb7c5254dc00538850b5d29d1ea985bb412f1583cc9f8f6dd641a205fc\": container with ID starting with 519ea8fb7c5254dc00538850b5d29d1ea985bb412f1583cc9f8f6dd641a205fc not found: ID does not exist" containerID="519ea8fb7c5254dc00538850b5d29d1ea985bb412f1583cc9f8f6dd641a205fc"
Sep 09 21:07:53.527127 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:07:53.527071    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"519ea8fb7c5254dc00538850b5d29d1ea985bb412f1583cc9f8f6dd641a205fc"} err="failed to get container status \"519ea8fb7c5254dc00538850b5d29d1ea985bb412f1583cc9f8f6dd641a205fc\": rpc error: code = NotFound desc = could not find container \"519ea8fb7c5254dc00538850b5d29d1ea985bb412f1583cc9f8f6dd641a205fc\": container with ID starting with 519ea8fb7c5254dc00538850b5d29d1ea985bb412f1583cc9f8f6dd641a205fc not found: ID does not exist"
Sep 09 21:07:53.527127 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:07:53.527090    2579 scope.go:117] "RemoveContainer" containerID="0dbc840ec11c33b39f98eb25dbdc59475746b5f300ca921a188d61c9bcfd25ae"
Sep 09 21:07:53.536091 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:07:53.536072    2579 scope.go:117] "RemoveContainer" containerID="a7d51b59e97a8fee5c7b22eadb7a27b36658d9b916c5dabc667304c2532d1b06"
Sep 09 21:07:53.544252 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:07:53.544232    2579 scope.go:117] "RemoveContainer" containerID="ab352443dd91393f9f9e996cf4933a784f6b659ba92127759ba70432b2b15bd0"
Sep 09 21:07:53.553204 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:07:53.553187    2579 scope.go:117] "RemoveContainer" containerID="0dbc840ec11c33b39f98eb25dbdc59475746b5f300ca921a188d61c9bcfd25ae"
Sep 09 21:07:53.553533 ip-10-0-138-224 kubenswrapper[2579]: E0909 21:07:53.553504    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0dbc840ec11c33b39f98eb25dbdc59475746b5f300ca921a188d61c9bcfd25ae\": container with ID starting with 0dbc840ec11c33b39f98eb25dbdc59475746b5f300ca921a188d61c9bcfd25ae not found: ID does not exist" containerID="0dbc840ec11c33b39f98eb25dbdc59475746b5f300ca921a188d61c9bcfd25ae"
Sep 09 21:07:53.553640 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:07:53.553543    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0dbc840ec11c33b39f98eb25dbdc59475746b5f300ca921a188d61c9bcfd25ae"} err="failed to get container status \"0dbc840ec11c33b39f98eb25dbdc59475746b5f300ca921a188d61c9bcfd25ae\": rpc error: code = NotFound desc = could not find container \"0dbc840ec11c33b39f98eb25dbdc59475746b5f300ca921a188d61c9bcfd25ae\": container with ID starting with 0dbc840ec11c33b39f98eb25dbdc59475746b5f300ca921a188d61c9bcfd25ae not found: ID does not exist"
Sep 09 21:07:53.553640 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:07:53.553566    2579 scope.go:117] "RemoveContainer" containerID="a7d51b59e97a8fee5c7b22eadb7a27b36658d9b916c5dabc667304c2532d1b06"
Sep 09 21:07:53.553863 ip-10-0-138-224 kubenswrapper[2579]: E0909 21:07:53.553840    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7d51b59e97a8fee5c7b22eadb7a27b36658d9b916c5dabc667304c2532d1b06\": container with ID starting with a7d51b59e97a8fee5c7b22eadb7a27b36658d9b916c5dabc667304c2532d1b06 not found: ID does not exist" containerID="a7d51b59e97a8fee5c7b22eadb7a27b36658d9b916c5dabc667304c2532d1b06"
Sep 09 21:07:53.553913 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:07:53.553865    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7d51b59e97a8fee5c7b22eadb7a27b36658d9b916c5dabc667304c2532d1b06"} err="failed to get container status \"a7d51b59e97a8fee5c7b22eadb7a27b36658d9b916c5dabc667304c2532d1b06\": rpc error: code = NotFound desc = could not find container \"a7d51b59e97a8fee5c7b22eadb7a27b36658d9b916c5dabc667304c2532d1b06\": container with ID starting with a7d51b59e97a8fee5c7b22eadb7a27b36658d9b916c5dabc667304c2532d1b06 not found: ID does not exist"
Sep 09 21:07:53.553913 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:07:53.553880    2579 scope.go:117] "RemoveContainer" containerID="ab352443dd91393f9f9e996cf4933a784f6b659ba92127759ba70432b2b15bd0"
Sep 09 21:07:53.554147 ip-10-0-138-224 kubenswrapper[2579]: E0909 21:07:53.554128    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab352443dd91393f9f9e996cf4933a784f6b659ba92127759ba70432b2b15bd0\": container with ID starting with ab352443dd91393f9f9e996cf4933a784f6b659ba92127759ba70432b2b15bd0 not found: ID does not exist" containerID="ab352443dd91393f9f9e996cf4933a784f6b659ba92127759ba70432b2b15bd0"
Sep 09 21:07:53.554203 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:07:53.554153    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab352443dd91393f9f9e996cf4933a784f6b659ba92127759ba70432b2b15bd0"} err="failed to get container status \"ab352443dd91393f9f9e996cf4933a784f6b659ba92127759ba70432b2b15bd0\": rpc error: code = NotFound desc = could not find container \"ab352443dd91393f9f9e996cf4933a784f6b659ba92127759ba70432b2b15bd0\": container with ID starting with ab352443dd91393f9f9e996cf4933a784f6b659ba92127759ba70432b2b15bd0 not found: ID does not exist"
Sep 09 21:07:53.579633 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:07:53.579604    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fcf465c7-1a79-47bd-9490-1dc8f09a5fca-kserve-provision-location\") pod \"fcf465c7-1a79-47bd-9490-1dc8f09a5fca\" (UID: \"fcf465c7-1a79-47bd-9490-1dc8f09a5fca\") "
Sep 09 21:07:53.579742 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:07:53.579645    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fcf465c7-1a79-47bd-9490-1dc8f09a5fca-proxy-tls\") pod \"fcf465c7-1a79-47bd-9490-1dc8f09a5fca\" (UID: \"fcf465c7-1a79-47bd-9490-1dc8f09a5fca\") "
Sep 09 21:07:53.579742 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:07:53.579673    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/26cfda75-30a0-41d6-87fe-f8466034a4d2-cabundle-cert\") pod \"26cfda75-30a0-41d6-87fe-f8466034a4d2\" (UID: \"26cfda75-30a0-41d6-87fe-f8466034a4d2\") "
Sep 09 21:07:53.579742 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:07:53.579701    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fcf465c7-1a79-47bd-9490-1dc8f09a5fca-kube-rbac-proxy-sar-config\") pod \"fcf465c7-1a79-47bd-9490-1dc8f09a5fca\" (UID: \"fcf465c7-1a79-47bd-9490-1dc8f09a5fca\") "
Sep 09 21:07:53.579925 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:07:53.579770    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/26cfda75-30a0-41d6-87fe-f8466034a4d2-kube-rbac-proxy-sar-config\") pod \"26cfda75-30a0-41d6-87fe-f8466034a4d2\" (UID: \"26cfda75-30a0-41d6-87fe-f8466034a4d2\") "
Sep 09 21:07:53.579925 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:07:53.579798    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fcf465c7-1a79-47bd-9490-1dc8f09a5fca-cabundle-cert\") pod \"fcf465c7-1a79-47bd-9490-1dc8f09a5fca\" (UID: \"fcf465c7-1a79-47bd-9490-1dc8f09a5fca\") "
Sep 09 21:07:53.579925 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:07:53.579823    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/26cfda75-30a0-41d6-87fe-f8466034a4d2-proxy-tls\") pod \"26cfda75-30a0-41d6-87fe-f8466034a4d2\" (UID: \"26cfda75-30a0-41d6-87fe-f8466034a4d2\") "
Sep 09 21:07:53.579925 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:07:53.579853    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4hc5p\" (UniqueName: \"kubernetes.io/projected/fcf465c7-1a79-47bd-9490-1dc8f09a5fca-kube-api-access-4hc5p\") pod \"fcf465c7-1a79-47bd-9490-1dc8f09a5fca\" (UID: \"fcf465c7-1a79-47bd-9490-1dc8f09a5fca\") "
Sep 09 21:07:53.579925 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:07:53.579881    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5pqzh\" (UniqueName: \"kubernetes.io/projected/26cfda75-30a0-41d6-87fe-f8466034a4d2-kube-api-access-5pqzh\") pod \"26cfda75-30a0-41d6-87fe-f8466034a4d2\" (UID: \"26cfda75-30a0-41d6-87fe-f8466034a4d2\") "
Sep 09 21:07:53.580156 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:07:53.579930    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/26cfda75-30a0-41d6-87fe-f8466034a4d2-kserve-provision-location\") pod \"26cfda75-30a0-41d6-87fe-f8466034a4d2\" (UID: \"26cfda75-30a0-41d6-87fe-f8466034a4d2\") "
Sep 09 21:07:53.580156 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:07:53.580023    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcf465c7-1a79-47bd-9490-1dc8f09a5fca-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "fcf465c7-1a79-47bd-9490-1dc8f09a5fca" (UID: "fcf465c7-1a79-47bd-9490-1dc8f09a5fca"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:07:53.580156 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:07:53.580044    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26cfda75-30a0-41d6-87fe-f8466034a4d2-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "26cfda75-30a0-41d6-87fe-f8466034a4d2" (UID: "26cfda75-30a0-41d6-87fe-f8466034a4d2"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:07:53.580156 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:07:53.580083    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fcf465c7-1a79-47bd-9490-1dc8f09a5fca-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "fcf465c7-1a79-47bd-9490-1dc8f09a5fca" (UID: "fcf465c7-1a79-47bd-9490-1dc8f09a5fca"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:07:53.580340 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:07:53.580321    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fcf465c7-1a79-47bd-9490-1dc8f09a5fca-kserve-provision-location\") on node \"ip-10-0-138-224.ec2.internal\" DevicePath \"\""
Sep 09 21:07:53.580399 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:07:53.580348    2579 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/26cfda75-30a0-41d6-87fe-f8466034a4d2-cabundle-cert\") on node \"ip-10-0-138-224.ec2.internal\" DevicePath \"\""
Sep 09 21:07:53.580399 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:07:53.580362    2579 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fcf465c7-1a79-47bd-9490-1dc8f09a5fca-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-224.ec2.internal\" DevicePath \"\""
Sep 09 21:07:53.580651 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:07:53.580613    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fcf465c7-1a79-47bd-9490-1dc8f09a5fca-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "fcf465c7-1a79-47bd-9490-1dc8f09a5fca" (UID: "fcf465c7-1a79-47bd-9490-1dc8f09a5fca"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:07:53.580705 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:07:53.580644    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26cfda75-30a0-41d6-87fe-f8466034a4d2-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "26cfda75-30a0-41d6-87fe-f8466034a4d2" (UID: "26cfda75-30a0-41d6-87fe-f8466034a4d2"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:07:53.580800 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:07:53.580763    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26cfda75-30a0-41d6-87fe-f8466034a4d2-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "26cfda75-30a0-41d6-87fe-f8466034a4d2" (UID: "26cfda75-30a0-41d6-87fe-f8466034a4d2"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:07:53.581990 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:07:53.581966    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fcf465c7-1a79-47bd-9490-1dc8f09a5fca-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "fcf465c7-1a79-47bd-9490-1dc8f09a5fca" (UID: "fcf465c7-1a79-47bd-9490-1dc8f09a5fca"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:07:53.582743 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:07:53.582711    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26cfda75-30a0-41d6-87fe-f8466034a4d2-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "26cfda75-30a0-41d6-87fe-f8466034a4d2" (UID: "26cfda75-30a0-41d6-87fe-f8466034a4d2"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:07:53.582917 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:07:53.582895    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcf465c7-1a79-47bd-9490-1dc8f09a5fca-kube-api-access-4hc5p" (OuterVolumeSpecName: "kube-api-access-4hc5p") pod "fcf465c7-1a79-47bd-9490-1dc8f09a5fca" (UID: "fcf465c7-1a79-47bd-9490-1dc8f09a5fca"). InnerVolumeSpecName "kube-api-access-4hc5p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:07:53.582917 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:07:53.582901    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26cfda75-30a0-41d6-87fe-f8466034a4d2-kube-api-access-5pqzh" (OuterVolumeSpecName: "kube-api-access-5pqzh") pod "26cfda75-30a0-41d6-87fe-f8466034a4d2" (UID: "26cfda75-30a0-41d6-87fe-f8466034a4d2"). InnerVolumeSpecName "kube-api-access-5pqzh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:07:53.681797 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:07:53.681759    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fcf465c7-1a79-47bd-9490-1dc8f09a5fca-proxy-tls\") on node \"ip-10-0-138-224.ec2.internal\" DevicePath \"\""
Sep 09 21:07:53.681797 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:07:53.681791    2579 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/26cfda75-30a0-41d6-87fe-f8466034a4d2-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-224.ec2.internal\" DevicePath \"\""
Sep 09 21:07:53.681797 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:07:53.681800    2579 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fcf465c7-1a79-47bd-9490-1dc8f09a5fca-cabundle-cert\") on node \"ip-10-0-138-224.ec2.internal\" DevicePath \"\""
Sep 09 21:07:53.681797 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:07:53.681809    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/26cfda75-30a0-41d6-87fe-f8466034a4d2-proxy-tls\") on node \"ip-10-0-138-224.ec2.internal\" DevicePath \"\""
Sep 09 21:07:53.682053 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:07:53.681819    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4hc5p\" (UniqueName: \"kubernetes.io/projected/fcf465c7-1a79-47bd-9490-1dc8f09a5fca-kube-api-access-4hc5p\") on node \"ip-10-0-138-224.ec2.internal\" DevicePath \"\""
Sep 09 21:07:53.682053 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:07:53.681827    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5pqzh\" (UniqueName: \"kubernetes.io/projected/26cfda75-30a0-41d6-87fe-f8466034a4d2-kube-api-access-5pqzh\") on node \"ip-10-0-138-224.ec2.internal\" DevicePath \"\""
Sep 09 21:07:53.682053 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:07:53.681835    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/26cfda75-30a0-41d6-87fe-f8466034a4d2-kserve-provision-location\") on node \"ip-10-0-138-224.ec2.internal\" DevicePath \"\""
Sep 09 21:07:53.823078 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:07:53.823046    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-8xkkd"]
Sep 09 21:07:53.827251 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:07:53.827225    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-8xkkd"]
Sep 09 21:07:53.838640 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:07:53.838617    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5bb8f495f4-ds7lm"]
Sep 09 21:07:53.842938 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:07:53.842916    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5bb8f495f4-ds7lm"]
Sep 09 21:07:55.290943 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:07:55.290912    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="26cfda75-30a0-41d6-87fe-f8466034a4d2" path="/var/lib/kubelet/pods/26cfda75-30a0-41d6-87fe-f8466034a4d2/volumes"
Sep 09 21:07:55.291394 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:07:55.291380    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fcf465c7-1a79-47bd-9490-1dc8f09a5fca" path="/var/lib/kubelet/pods/fcf465c7-1a79-47bd-9490-1dc8f09a5fca/volumes"
Sep 09 21:08:01.175216 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:08:01.175185    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-887b3-predictor-5ddcd6d77d-gs5gp"
Sep 09 21:08:18.205793 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:08:18.205755    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-hpa-887b3-86d9b7b5c8-fhrvh"]
Sep 09 21:08:18.208190 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:08:18.206288    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26cfda75-30a0-41d6-87fe-f8466034a4d2" containerName="storage-initializer"
Sep 09 21:08:18.208190 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:08:18.206306    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="26cfda75-30a0-41d6-87fe-f8466034a4d2" containerName="storage-initializer"
Sep 09 21:08:18.208190 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:08:18.206315    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26cfda75-30a0-41d6-87fe-f8466034a4d2" containerName="kserve-container"
Sep 09 21:08:18.208190 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:08:18.206321    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="26cfda75-30a0-41d6-87fe-f8466034a4d2" containerName="kserve-container"
Sep 09 21:08:18.208190 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:08:18.206331    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a25dae2-a07b-4d1b-a439-15b2bd1b2915" containerName="model-chainer-raw-c959c"
Sep 09 21:08:18.208190 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:08:18.206336    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a25dae2-a07b-4d1b-a439-15b2bd1b2915" containerName="model-chainer-raw-c959c"
Sep 09 21:08:18.208190 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:08:18.206345    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fcf465c7-1a79-47bd-9490-1dc8f09a5fca" containerName="storage-initializer"
Sep 09 21:08:18.208190 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:08:18.206350    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="fcf465c7-1a79-47bd-9490-1dc8f09a5fca" containerName="storage-initializer"
Sep 09 21:08:18.208190 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:08:18.206360    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26cfda75-30a0-41d6-87fe-f8466034a4d2" containerName="kube-rbac-proxy"
Sep 09 21:08:18.208190 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:08:18.206365    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="26cfda75-30a0-41d6-87fe-f8466034a4d2" containerName="kube-rbac-proxy"
Sep 09 21:08:18.208190 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:08:18.206373    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fcf465c7-1a79-47bd-9490-1dc8f09a5fca" containerName="kserve-container"
Sep 09 21:08:18.208190 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:08:18.206378    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="fcf465c7-1a79-47bd-9490-1dc8f09a5fca" containerName="kserve-container"
Sep 09 21:08:18.208190 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:08:18.206390    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fcf465c7-1a79-47bd-9490-1dc8f09a5fca" containerName="kube-rbac-proxy"
Sep 09 21:08:18.208190 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:08:18.206395    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="fcf465c7-1a79-47bd-9490-1dc8f09a5fca" containerName="kube-rbac-proxy"
Sep 09 21:08:18.208190 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:08:18.206466    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="26cfda75-30a0-41d6-87fe-f8466034a4d2" containerName="kserve-container"
Sep 09 21:08:18.208190 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:08:18.206474    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="fcf465c7-1a79-47bd-9490-1dc8f09a5fca" containerName="kserve-container"
Sep 09 21:08:18.208190 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:08:18.206482    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="26cfda75-30a0-41d6-87fe-f8466034a4d2" containerName="kube-rbac-proxy"
Sep 09 21:08:18.208190 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:08:18.206488    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="fcf465c7-1a79-47bd-9490-1dc8f09a5fca" containerName="kube-rbac-proxy"
Sep 09 21:08:18.208190 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:08:18.206497    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="0a25dae2-a07b-4d1b-a439-15b2bd1b2915" containerName="model-chainer-raw-c959c"
Sep 09 21:08:18.209323 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:08:18.209303    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-887b3-86d9b7b5c8-fhrvh"
Sep 09 21:08:18.212100 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:08:18.212074    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"model-chainer-raw-hpa-887b3-kube-rbac-proxy-sar-config\""
Sep 09 21:08:18.212218 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:08:18.212139    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"model-chainer-raw-hpa-887b3-serving-cert\""
Sep 09 21:08:18.220359 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:08:18.220338    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-hpa-887b3-86d9b7b5c8-fhrvh"]
Sep 09 21:08:18.288061 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:08:18.288031    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0c7d0bde-a4ca-49d9-a084-f646b52d63f9-openshift-service-ca-bundle\") pod \"model-chainer-raw-hpa-887b3-86d9b7b5c8-fhrvh\" (UID: \"0c7d0bde-a4ca-49d9-a084-f646b52d63f9\") " pod="kserve-ci-e2e-test/model-chainer-raw-hpa-887b3-86d9b7b5c8-fhrvh"
Sep 09 21:08:18.288161 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:08:18.288064    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0c7d0bde-a4ca-49d9-a084-f646b52d63f9-proxy-tls\") pod \"model-chainer-raw-hpa-887b3-86d9b7b5c8-fhrvh\" (UID: \"0c7d0bde-a4ca-49d9-a084-f646b52d63f9\") " pod="kserve-ci-e2e-test/model-chainer-raw-hpa-887b3-86d9b7b5c8-fhrvh"
Sep 09 21:08:18.389492 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:08:18.389459    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0c7d0bde-a4ca-49d9-a084-f646b52d63f9-openshift-service-ca-bundle\") pod \"model-chainer-raw-hpa-887b3-86d9b7b5c8-fhrvh\" (UID: \"0c7d0bde-a4ca-49d9-a084-f646b52d63f9\") " pod="kserve-ci-e2e-test/model-chainer-raw-hpa-887b3-86d9b7b5c8-fhrvh"
Sep 09 21:08:18.389630 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:08:18.389500    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0c7d0bde-a4ca-49d9-a084-f646b52d63f9-proxy-tls\") pod \"model-chainer-raw-hpa-887b3-86d9b7b5c8-fhrvh\" (UID: \"0c7d0bde-a4ca-49d9-a084-f646b52d63f9\") " pod="kserve-ci-e2e-test/model-chainer-raw-hpa-887b3-86d9b7b5c8-fhrvh"
Sep 09 21:08:18.389630 ip-10-0-138-224 kubenswrapper[2579]: E0909 21:08:18.389625    2579 secret.go:189] Couldn't get secret kserve-ci-e2e-test/model-chainer-raw-hpa-887b3-serving-cert: secret "model-chainer-raw-hpa-887b3-serving-cert" not found
Sep 09 21:08:18.389704 ip-10-0-138-224 kubenswrapper[2579]: E0909 21:08:18.389687    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0c7d0bde-a4ca-49d9-a084-f646b52d63f9-proxy-tls podName:0c7d0bde-a4ca-49d9-a084-f646b52d63f9 nodeName:}" failed. No retries permitted until 2026-09-09 21:08:18.889670787 +0000 UTC m=+956.129764888 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/0c7d0bde-a4ca-49d9-a084-f646b52d63f9-proxy-tls") pod "model-chainer-raw-hpa-887b3-86d9b7b5c8-fhrvh" (UID: "0c7d0bde-a4ca-49d9-a084-f646b52d63f9") : secret "model-chainer-raw-hpa-887b3-serving-cert" not found
Sep 09 21:08:18.390014 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:08:18.389994    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0c7d0bde-a4ca-49d9-a084-f646b52d63f9-openshift-service-ca-bundle\") pod \"model-chainer-raw-hpa-887b3-86d9b7b5c8-fhrvh\" (UID: \"0c7d0bde-a4ca-49d9-a084-f646b52d63f9\") " pod="kserve-ci-e2e-test/model-chainer-raw-hpa-887b3-86d9b7b5c8-fhrvh"
Sep 09 21:08:18.893749 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:08:18.893717    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0c7d0bde-a4ca-49d9-a084-f646b52d63f9-proxy-tls\") pod \"model-chainer-raw-hpa-887b3-86d9b7b5c8-fhrvh\" (UID: \"0c7d0bde-a4ca-49d9-a084-f646b52d63f9\") " pod="kserve-ci-e2e-test/model-chainer-raw-hpa-887b3-86d9b7b5c8-fhrvh"
Sep 09 21:08:18.896292 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:08:18.896271    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0c7d0bde-a4ca-49d9-a084-f646b52d63f9-proxy-tls\") pod \"model-chainer-raw-hpa-887b3-86d9b7b5c8-fhrvh\" (UID: \"0c7d0bde-a4ca-49d9-a084-f646b52d63f9\") " pod="kserve-ci-e2e-test/model-chainer-raw-hpa-887b3-86d9b7b5c8-fhrvh"
Sep 09 21:08:19.120117 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:08:19.120084    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-887b3-86d9b7b5c8-fhrvh"
Sep 09 21:08:19.395907 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:08:19.395883    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-hpa-887b3-86d9b7b5c8-fhrvh"]
Sep 09 21:08:19.398291 ip-10-0-138-224 kubenswrapper[2579]: W0909 21:08:19.398264    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c7d0bde_a4ca_49d9_a084_f646b52d63f9.slice/crio-7e010ea4a12309cd43d7fb02d080372e831a28a8c343e6c0021942bba46b84c5 WatchSource:0}: Error finding container 7e010ea4a12309cd43d7fb02d080372e831a28a8c343e6c0021942bba46b84c5: Status 404 returned error can't find the container with id 7e010ea4a12309cd43d7fb02d080372e831a28a8c343e6c0021942bba46b84c5
Sep 09 21:08:19.622767 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:08:19.622731    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-887b3-86d9b7b5c8-fhrvh" event={"ID":"0c7d0bde-a4ca-49d9-a084-f646b52d63f9","Type":"ContainerStarted","Data":"39113329548a81dce645d9b8d1d7b3e0b49f9cd33500c034f98b3d458e8359a9"}
Sep 09 21:08:19.622938 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:08:19.622794    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-887b3-86d9b7b5c8-fhrvh" event={"ID":"0c7d0bde-a4ca-49d9-a084-f646b52d63f9","Type":"ContainerStarted","Data":"7e010ea4a12309cd43d7fb02d080372e831a28a8c343e6c0021942bba46b84c5"}
Sep 09 21:08:19.622938 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:08:19.622872    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-887b3-86d9b7b5c8-fhrvh"
Sep 09 21:08:19.640207 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:08:19.640161    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-887b3-86d9b7b5c8-fhrvh" podStartSLOduration=1.6401448429999999 podStartE2EDuration="1.640144843s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:08:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:08:19.639110224 +0000 UTC m=+956.879204355" watchObservedRunningTime="2026-09-09 21:08:19.640144843 +0000 UTC m=+956.880238961"
Sep 09 21:08:25.634419 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:08:25.634387    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-887b3-86d9b7b5c8-fhrvh"
Sep 09 21:08:28.281843 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:08:28.281813    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-hpa-887b3-86d9b7b5c8-fhrvh"]
Sep 09 21:08:28.282203 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:08:28.282013    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-887b3-86d9b7b5c8-fhrvh" podUID="0c7d0bde-a4ca-49d9-a084-f646b52d63f9" containerName="model-chainer-raw-hpa-887b3" containerID="cri-o://39113329548a81dce645d9b8d1d7b3e0b49f9cd33500c034f98b3d458e8359a9" gracePeriod=30
Sep 09 21:08:28.397829 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:08:28.397796    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-887b3-predictor-5ddcd6d77d-gs5gp"]
Sep 09 21:08:28.398122 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:08:28.398086    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-887b3-predictor-5ddcd6d77d-gs5gp" podUID="c0ff6dd4-d542-4467-8222-702149441465" containerName="kserve-container" containerID="cri-o://4be51b774b82a0473bce1a544c428c3cc718a1aaaef67f927a8f3300ba30e629" gracePeriod=30
Sep 09 21:08:28.398268 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:08:28.398133    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-887b3-predictor-5ddcd6d77d-gs5gp" podUID="c0ff6dd4-d542-4467-8222-702149441465" containerName="kube-rbac-proxy" containerID="cri-o://9dd50d2048a6ddea512db058809656e04cecb8470ef6a3c44eaec16fdb5ab2a1" gracePeriod=30
Sep 09 21:08:28.514931 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:08:28.514899    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-afe12-predictor-57759dbb49-s4j9t"]
Sep 09 21:08:28.518224 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:08:28.518205    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-afe12-predictor-57759dbb49-s4j9t"
Sep 09 21:08:28.520864 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:08:28.520845    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-raw-afe12-kube-rbac-proxy-sar-config\""
Sep 09 21:08:28.520963 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:08:28.520846    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-raw-afe12-predictor-serving-cert\""
Sep 09 21:08:28.531727 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:08:28.531705    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-afe12-predictor-57759dbb49-s4j9t"]
Sep 09 21:08:28.663997 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:08:28.663963    2579 generic.go:356] "Generic (PLEG): container finished" podID="c0ff6dd4-d542-4467-8222-702149441465" containerID="9dd50d2048a6ddea512db058809656e04cecb8470ef6a3c44eaec16fdb5ab2a1" exitCode=2
Sep 09 21:08:28.664145 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:08:28.664044    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-887b3-predictor-5ddcd6d77d-gs5gp" event={"ID":"c0ff6dd4-d542-4467-8222-702149441465","Type":"ContainerDied","Data":"9dd50d2048a6ddea512db058809656e04cecb8470ef6a3c44eaec16fdb5ab2a1"}
Sep 09 21:08:28.677359 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:08:28.677333    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e722d823-93a6-4495-9e58-0407ab0d8fd5-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-afe12-predictor-57759dbb49-s4j9t\" (UID: \"e722d823-93a6-4495-9e58-0407ab0d8fd5\") " pod="kserve-ci-e2e-test/message-dumper-raw-afe12-predictor-57759dbb49-s4j9t"
Sep 09 21:08:28.677492 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:08:28.677369    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2r7pm\" (UniqueName: \"kubernetes.io/projected/e722d823-93a6-4495-9e58-0407ab0d8fd5-kube-api-access-2r7pm\") pod \"message-dumper-raw-afe12-predictor-57759dbb49-s4j9t\" (UID: \"e722d823-93a6-4495-9e58-0407ab0d8fd5\") " pod="kserve-ci-e2e-test/message-dumper-raw-afe12-predictor-57759dbb49-s4j9t"
Sep 09 21:08:28.677492 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:08:28.677389    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e722d823-93a6-4495-9e58-0407ab0d8fd5-proxy-tls\") pod \"message-dumper-raw-afe12-predictor-57759dbb49-s4j9t\" (UID: \"e722d823-93a6-4495-9e58-0407ab0d8fd5\") " pod="kserve-ci-e2e-test/message-dumper-raw-afe12-predictor-57759dbb49-s4j9t"
Sep 09 21:08:28.778500 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:08:28.778475    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2r7pm\" (UniqueName: \"kubernetes.io/projected/e722d823-93a6-4495-9e58-0407ab0d8fd5-kube-api-access-2r7pm\") pod \"message-dumper-raw-afe12-predictor-57759dbb49-s4j9t\" (UID: \"e722d823-93a6-4495-9e58-0407ab0d8fd5\") " pod="kserve-ci-e2e-test/message-dumper-raw-afe12-predictor-57759dbb49-s4j9t"
Sep 09 21:08:28.778615 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:08:28.778505    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e722d823-93a6-4495-9e58-0407ab0d8fd5-proxy-tls\") pod \"message-dumper-raw-afe12-predictor-57759dbb49-s4j9t\" (UID: \"e722d823-93a6-4495-9e58-0407ab0d8fd5\") " pod="kserve-ci-e2e-test/message-dumper-raw-afe12-predictor-57759dbb49-s4j9t"
Sep 09 21:08:28.778615 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:08:28.778573    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e722d823-93a6-4495-9e58-0407ab0d8fd5-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-afe12-predictor-57759dbb49-s4j9t\" (UID: \"e722d823-93a6-4495-9e58-0407ab0d8fd5\") " pod="kserve-ci-e2e-test/message-dumper-raw-afe12-predictor-57759dbb49-s4j9t"
Sep 09 21:08:28.778734 ip-10-0-138-224 kubenswrapper[2579]: E0909 21:08:28.778703    2579 secret.go:189] Couldn't get secret kserve-ci-e2e-test/message-dumper-raw-afe12-predictor-serving-cert: secret "message-dumper-raw-afe12-predictor-serving-cert" not found
Sep 09 21:08:28.778791 ip-10-0-138-224 kubenswrapper[2579]: E0909 21:08:28.778782    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e722d823-93a6-4495-9e58-0407ab0d8fd5-proxy-tls podName:e722d823-93a6-4495-9e58-0407ab0d8fd5 nodeName:}" failed. No retries permitted until 2026-09-09 21:08:29.278762499 +0000 UTC m=+966.518856609 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/e722d823-93a6-4495-9e58-0407ab0d8fd5-proxy-tls") pod "message-dumper-raw-afe12-predictor-57759dbb49-s4j9t" (UID: "e722d823-93a6-4495-9e58-0407ab0d8fd5") : secret "message-dumper-raw-afe12-predictor-serving-cert" not found
Sep 09 21:08:28.779133 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:08:28.779113    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e722d823-93a6-4495-9e58-0407ab0d8fd5-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-afe12-predictor-57759dbb49-s4j9t\" (UID: \"e722d823-93a6-4495-9e58-0407ab0d8fd5\") " pod="kserve-ci-e2e-test/message-dumper-raw-afe12-predictor-57759dbb49-s4j9t"
Sep 09 21:08:28.789129 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:08:28.789109    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2r7pm\" (UniqueName: \"kubernetes.io/projected/e722d823-93a6-4495-9e58-0407ab0d8fd5-kube-api-access-2r7pm\") pod \"message-dumper-raw-afe12-predictor-57759dbb49-s4j9t\" (UID: \"e722d823-93a6-4495-9e58-0407ab0d8fd5\") " pod="kserve-ci-e2e-test/message-dumper-raw-afe12-predictor-57759dbb49-s4j9t"
Sep 09 21:08:29.283187 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:08:29.283157    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e722d823-93a6-4495-9e58-0407ab0d8fd5-proxy-tls\") pod \"message-dumper-raw-afe12-predictor-57759dbb49-s4j9t\" (UID: \"e722d823-93a6-4495-9e58-0407ab0d8fd5\") " pod="kserve-ci-e2e-test/message-dumper-raw-afe12-predictor-57759dbb49-s4j9t"
Sep 09 21:08:29.285853 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:08:29.285817    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e722d823-93a6-4495-9e58-0407ab0d8fd5-proxy-tls\") pod \"message-dumper-raw-afe12-predictor-57759dbb49-s4j9t\" (UID: \"e722d823-93a6-4495-9e58-0407ab0d8fd5\") " pod="kserve-ci-e2e-test/message-dumper-raw-afe12-predictor-57759dbb49-s4j9t"
Sep 09 21:08:29.428918 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:08:29.428888    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-afe12-predictor-57759dbb49-s4j9t"
Sep 09 21:08:29.700176 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:08:29.700153    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-afe12-predictor-57759dbb49-s4j9t"]
Sep 09 21:08:29.702560 ip-10-0-138-224 kubenswrapper[2579]: W0909 21:08:29.702526    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode722d823_93a6_4495_9e58_0407ab0d8fd5.slice/crio-98c404454f0b6d1a90b1ab2a077403b6197e53a98d3e1ff9976e2919c03a2c4f WatchSource:0}: Error finding container 98c404454f0b6d1a90b1ab2a077403b6197e53a98d3e1ff9976e2919c03a2c4f: Status 404 returned error can't find the container with id 98c404454f0b6d1a90b1ab2a077403b6197e53a98d3e1ff9976e2919c03a2c4f
Sep 09 21:08:29.704649 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:08:29.704629    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 21:08:30.632594 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:08:30.632560    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-887b3-86d9b7b5c8-fhrvh" podUID="0c7d0bde-a4ca-49d9-a084-f646b52d63f9" containerName="model-chainer-raw-hpa-887b3" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 21:08:30.673798 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:08:30.673760    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-afe12-predictor-57759dbb49-s4j9t" event={"ID":"e722d823-93a6-4495-9e58-0407ab0d8fd5","Type":"ContainerStarted","Data":"98c404454f0b6d1a90b1ab2a077403b6197e53a98d3e1ff9976e2919c03a2c4f"}
Sep 09 21:08:31.169800 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:08:31.169764    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-887b3-predictor-5ddcd6d77d-gs5gp" podUID="c0ff6dd4-d542-4467-8222-702149441465" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.36:8643/healthz\": dial tcp 10.134.0.36:8643: connect: connection refused"
Sep 09 21:08:31.174063 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:08:31.174039    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-887b3-predictor-5ddcd6d77d-gs5gp" podUID="c0ff6dd4-d542-4467-8222-702149441465" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 09 21:08:31.679772 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:08:31.679730    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-afe12-predictor-57759dbb49-s4j9t" event={"ID":"e722d823-93a6-4495-9e58-0407ab0d8fd5","Type":"ContainerStarted","Data":"cab9bc58008dcaedde5054741521a50b2d36a09e5f5ec653e16318451c05d771"}
Sep 09 21:08:31.680191 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:08:31.679804    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-afe12-predictor-57759dbb49-s4j9t" event={"ID":"e722d823-93a6-4495-9e58-0407ab0d8fd5","Type":"ContainerStarted","Data":"91f6c681920fd47dc1ad558b8355ec371b6cf5c0bcb7d7dfad4188eb8f7de5ef"}
Sep 09 21:08:31.680191 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:08:31.679947    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-raw-afe12-predictor-57759dbb49-s4j9t"
Sep 09 21:08:31.680191 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:08:31.680088    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-raw-afe12-predictor-57759dbb49-s4j9t"
Sep 09 21:08:31.681823 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:08:31.681804    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-raw-afe12-predictor-57759dbb49-s4j9t"
Sep 09 21:08:31.702400 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:08:31.702326    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/message-dumper-raw-afe12-predictor-57759dbb49-s4j9t" podStartSLOduration=2.625357739 podStartE2EDuration="3.702312206s" totalImagesPullingTime="1.076954467s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:08:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:08:31.70081948 +0000 UTC m=+968.940913599" watchObservedRunningTime="2026-09-09 21:08:31.702312206 +0000 UTC m=+968.942406324"
Sep 09 21:08:32.647011 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:08:32.646986    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-887b3-predictor-5ddcd6d77d-gs5gp"
Sep 09 21:08:32.686380 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:08:32.686343    2579 generic.go:356] "Generic (PLEG): container finished" podID="c0ff6dd4-d542-4467-8222-702149441465" containerID="4be51b774b82a0473bce1a544c428c3cc718a1aaaef67f927a8f3300ba30e629" exitCode=0
Sep 09 21:08:32.686773 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:08:32.686421    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-887b3-predictor-5ddcd6d77d-gs5gp"
Sep 09 21:08:32.686773 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:08:32.686422    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-887b3-predictor-5ddcd6d77d-gs5gp" event={"ID":"c0ff6dd4-d542-4467-8222-702149441465","Type":"ContainerDied","Data":"4be51b774b82a0473bce1a544c428c3cc718a1aaaef67f927a8f3300ba30e629"}
Sep 09 21:08:32.686773 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:08:32.686537    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-887b3-predictor-5ddcd6d77d-gs5gp" event={"ID":"c0ff6dd4-d542-4467-8222-702149441465","Type":"ContainerDied","Data":"1daae64f2c02a2be2609e54e3f93ba268fa75d4174fc71d2676080d01421859a"}
Sep 09 21:08:32.686773 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:08:32.686566    2579 scope.go:117] "RemoveContainer" containerID="9dd50d2048a6ddea512db058809656e04cecb8470ef6a3c44eaec16fdb5ab2a1"
Sep 09 21:08:32.695460 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:08:32.695422    2579 scope.go:117] "RemoveContainer" containerID="4be51b774b82a0473bce1a544c428c3cc718a1aaaef67f927a8f3300ba30e629"
Sep 09 21:08:32.702859 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:08:32.702832    2579 scope.go:117] "RemoveContainer" containerID="6ed37560c5c70358baa1b4db9b6400db61a823b8f088c2c4bb6fade5f2999b22"
Sep 09 21:08:32.709831 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:08:32.709814    2579 scope.go:117] "RemoveContainer" containerID="9dd50d2048a6ddea512db058809656e04cecb8470ef6a3c44eaec16fdb5ab2a1"
Sep 09 21:08:32.710048 ip-10-0-138-224 kubenswrapper[2579]: E0909 21:08:32.710030    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9dd50d2048a6ddea512db058809656e04cecb8470ef6a3c44eaec16fdb5ab2a1\": container with ID starting with 9dd50d2048a6ddea512db058809656e04cecb8470ef6a3c44eaec16fdb5ab2a1 not found: ID does not exist" containerID="9dd50d2048a6ddea512db058809656e04cecb8470ef6a3c44eaec16fdb5ab2a1"
Sep 09 21:08:32.710103 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:08:32.710056    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9dd50d2048a6ddea512db058809656e04cecb8470ef6a3c44eaec16fdb5ab2a1"} err="failed to get container status \"9dd50d2048a6ddea512db058809656e04cecb8470ef6a3c44eaec16fdb5ab2a1\": rpc error: code = NotFound desc = could not find container \"9dd50d2048a6ddea512db058809656e04cecb8470ef6a3c44eaec16fdb5ab2a1\": container with ID starting with 9dd50d2048a6ddea512db058809656e04cecb8470ef6a3c44eaec16fdb5ab2a1 not found: ID does not exist"
Sep 09 21:08:32.710103 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:08:32.710073    2579 scope.go:117] "RemoveContainer" containerID="4be51b774b82a0473bce1a544c428c3cc718a1aaaef67f927a8f3300ba30e629"
Sep 09 21:08:32.710268 ip-10-0-138-224 kubenswrapper[2579]: E0909 21:08:32.710253    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4be51b774b82a0473bce1a544c428c3cc718a1aaaef67f927a8f3300ba30e629\": container with ID starting with 4be51b774b82a0473bce1a544c428c3cc718a1aaaef67f927a8f3300ba30e629 not found: ID does not exist" containerID="4be51b774b82a0473bce1a544c428c3cc718a1aaaef67f927a8f3300ba30e629"
Sep 09 21:08:32.710312 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:08:32.710272    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4be51b774b82a0473bce1a544c428c3cc718a1aaaef67f927a8f3300ba30e629"} err="failed to get container status \"4be51b774b82a0473bce1a544c428c3cc718a1aaaef67f927a8f3300ba30e629\": rpc error: code = NotFound desc = could not find container \"4be51b774b82a0473bce1a544c428c3cc718a1aaaef67f927a8f3300ba30e629\": container with ID starting with 4be51b774b82a0473bce1a544c428c3cc718a1aaaef67f927a8f3300ba30e629 not found: ID does not exist"
Sep 09 21:08:32.710312 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:08:32.710284    2579 scope.go:117] "RemoveContainer" containerID="6ed37560c5c70358baa1b4db9b6400db61a823b8f088c2c4bb6fade5f2999b22"
Sep 09 21:08:32.710617 ip-10-0-138-224 kubenswrapper[2579]: E0909 21:08:32.710598    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ed37560c5c70358baa1b4db9b6400db61a823b8f088c2c4bb6fade5f2999b22\": container with ID starting with 6ed37560c5c70358baa1b4db9b6400db61a823b8f088c2c4bb6fade5f2999b22 not found: ID does not exist" containerID="6ed37560c5c70358baa1b4db9b6400db61a823b8f088c2c4bb6fade5f2999b22"
Sep 09 21:08:32.710673 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:08:32.710622    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ed37560c5c70358baa1b4db9b6400db61a823b8f088c2c4bb6fade5f2999b22"} err="failed to get container status \"6ed37560c5c70358baa1b4db9b6400db61a823b8f088c2c4bb6fade5f2999b22\": rpc error: code = NotFound desc = could not find container \"6ed37560c5c70358baa1b4db9b6400db61a823b8f088c2c4bb6fade5f2999b22\": container with ID starting with 6ed37560c5c70358baa1b4db9b6400db61a823b8f088c2c4bb6fade5f2999b22 not found: ID does not exist"
Sep 09 21:08:32.814551 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:08:32.814473    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c0ff6dd4-d542-4467-8222-702149441465-kserve-provision-location\") pod \"c0ff6dd4-d542-4467-8222-702149441465\" (UID: \"c0ff6dd4-d542-4467-8222-702149441465\") "
Sep 09 21:08:32.814551 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:08:32.814514    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c0ff6dd4-d542-4467-8222-702149441465-kube-rbac-proxy-sar-config\") pod \"c0ff6dd4-d542-4467-8222-702149441465\" (UID: \"c0ff6dd4-d542-4467-8222-702149441465\") "
Sep 09 21:08:32.814551 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:08:32.814542    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bzblt\" (UniqueName: \"kubernetes.io/projected/c0ff6dd4-d542-4467-8222-702149441465-kube-api-access-bzblt\") pod \"c0ff6dd4-d542-4467-8222-702149441465\" (UID: \"c0ff6dd4-d542-4467-8222-702149441465\") "
Sep 09 21:08:32.814794 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:08:32.814610    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c0ff6dd4-d542-4467-8222-702149441465-cabundle-cert\") pod \"c0ff6dd4-d542-4467-8222-702149441465\" (UID: \"c0ff6dd4-d542-4467-8222-702149441465\") "
Sep 09 21:08:32.814794 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:08:32.814681    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c0ff6dd4-d542-4467-8222-702149441465-proxy-tls\") pod \"c0ff6dd4-d542-4467-8222-702149441465\" (UID: \"c0ff6dd4-d542-4467-8222-702149441465\") "
Sep 09 21:08:32.814912 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:08:32.814817    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0ff6dd4-d542-4467-8222-702149441465-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c0ff6dd4-d542-4467-8222-702149441465" (UID: "c0ff6dd4-d542-4467-8222-702149441465"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:08:32.814976 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:08:32.814852    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0ff6dd4-d542-4467-8222-702149441465-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "c0ff6dd4-d542-4467-8222-702149441465" (UID: "c0ff6dd4-d542-4467-8222-702149441465"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:08:32.815038 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:08:32.814993    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c0ff6dd4-d542-4467-8222-702149441465-kserve-provision-location\") on node \"ip-10-0-138-224.ec2.internal\" DevicePath \"\""
Sep 09 21:08:32.815087 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:08:32.815035    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0ff6dd4-d542-4467-8222-702149441465-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "c0ff6dd4-d542-4467-8222-702149441465" (UID: "c0ff6dd4-d542-4467-8222-702149441465"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:08:32.816872 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:08:32.816843    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0ff6dd4-d542-4467-8222-702149441465-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c0ff6dd4-d542-4467-8222-702149441465" (UID: "c0ff6dd4-d542-4467-8222-702149441465"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:08:32.816872 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:08:32.816863    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0ff6dd4-d542-4467-8222-702149441465-kube-api-access-bzblt" (OuterVolumeSpecName: "kube-api-access-bzblt") pod "c0ff6dd4-d542-4467-8222-702149441465" (UID: "c0ff6dd4-d542-4467-8222-702149441465"). InnerVolumeSpecName "kube-api-access-bzblt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:08:32.916160 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:08:32.916132    2579 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c0ff6dd4-d542-4467-8222-702149441465-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-224.ec2.internal\" DevicePath \"\""
Sep 09 21:08:32.916160 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:08:32.916154    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bzblt\" (UniqueName: \"kubernetes.io/projected/c0ff6dd4-d542-4467-8222-702149441465-kube-api-access-bzblt\") on node \"ip-10-0-138-224.ec2.internal\" DevicePath \"\""
Sep 09 21:08:32.916289 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:08:32.916164    2579 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c0ff6dd4-d542-4467-8222-702149441465-cabundle-cert\") on node \"ip-10-0-138-224.ec2.internal\" DevicePath \"\""
Sep 09 21:08:32.916289 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:08:32.916174    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c0ff6dd4-d542-4467-8222-702149441465-proxy-tls\") on node \"ip-10-0-138-224.ec2.internal\" DevicePath \"\""
Sep 09 21:08:33.007300 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:08:33.007276    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-887b3-predictor-5ddcd6d77d-gs5gp"]
Sep 09 21:08:33.010682 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:08:33.010660    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-887b3-predictor-5ddcd6d77d-gs5gp"]
Sep 09 21:08:33.291259 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:08:33.291233    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c0ff6dd4-d542-4467-8222-702149441465" path="/var/lib/kubelet/pods/c0ff6dd4-d542-4467-8222-702149441465/volumes"
Sep 09 21:08:35.631943 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:08:35.631907    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-887b3-86d9b7b5c8-fhrvh" podUID="0c7d0bde-a4ca-49d9-a084-f646b52d63f9" containerName="model-chainer-raw-hpa-887b3" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 21:08:38.697315 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:08:38.697289    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-raw-afe12-predictor-57759dbb49-s4j9t"
Sep 09 21:08:40.631235 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:08:40.631195    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-887b3-86d9b7b5c8-fhrvh" podUID="0c7d0bde-a4ca-49d9-a084-f646b52d63f9" containerName="model-chainer-raw-hpa-887b3" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 21:08:40.631624 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:08:40.631297    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-887b3-86d9b7b5c8-fhrvh"
Sep 09 21:08:45.631385 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:08:45.631304    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-887b3-86d9b7b5c8-fhrvh" podUID="0c7d0bde-a4ca-49d9-a084-f646b52d63f9" containerName="model-chainer-raw-hpa-887b3" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 21:08:48.541298 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:08:48.541264    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-afe12-predictor-849d5f794b-6nl27"]
Sep 09 21:08:48.541718 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:08:48.541642    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c0ff6dd4-d542-4467-8222-702149441465" containerName="kube-rbac-proxy"
Sep 09 21:08:48.541718 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:08:48.541654    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="c0ff6dd4-d542-4467-8222-702149441465" containerName="kube-rbac-proxy"
Sep 09 21:08:48.541718 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:08:48.541670    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c0ff6dd4-d542-4467-8222-702149441465" containerName="storage-initializer"
Sep 09 21:08:48.541718 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:08:48.541677    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="c0ff6dd4-d542-4467-8222-702149441465" containerName="storage-initializer"
Sep 09 21:08:48.541718 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:08:48.541686    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c0ff6dd4-d542-4467-8222-702149441465" containerName="kserve-container"
Sep 09 21:08:48.541718 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:08:48.541692    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="c0ff6dd4-d542-4467-8222-702149441465" containerName="kserve-container"
Sep 09 21:08:48.541931 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:08:48.541749    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="c0ff6dd4-d542-4467-8222-702149441465" containerName="kube-rbac-proxy"
Sep 09 21:08:48.541931 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:08:48.541757    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="c0ff6dd4-d542-4467-8222-702149441465" containerName="kserve-container"
Sep 09 21:08:48.546773 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:08:48.546749    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-afe12-predictor-849d5f794b-6nl27"
Sep 09 21:08:48.549732 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:08:48.549712    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-raw-afe12-kube-rbac-proxy-sar-config\""
Sep 09 21:08:48.549870 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:08:48.549712    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 09 21:08:48.549980 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:08:48.549962    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-raw-afe12-predictor-serving-cert\""
Sep 09 21:08:48.550191 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:08:48.550174    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 09 21:08:48.564018 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:08:48.563999    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-afe12-predictor-849d5f794b-6nl27"]
Sep 09 21:08:48.631033 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:08:48.631000    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/493338d2-24b1-47aa-a365-91e4d5274121-kserve-provision-location\") pod \"isvc-logger-raw-afe12-predictor-849d5f794b-6nl27\" (UID: \"493338d2-24b1-47aa-a365-91e4d5274121\") " pod="kserve-ci-e2e-test/isvc-logger-raw-afe12-predictor-849d5f794b-6nl27"
Sep 09 21:08:48.631159 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:08:48.631104    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mkrp5\" (UniqueName: \"kubernetes.io/projected/493338d2-24b1-47aa-a365-91e4d5274121-kube-api-access-mkrp5\") pod \"isvc-logger-raw-afe12-predictor-849d5f794b-6nl27\" (UID: \"493338d2-24b1-47aa-a365-91e4d5274121\") " pod="kserve-ci-e2e-test/isvc-logger-raw-afe12-predictor-849d5f794b-6nl27"
Sep 09 21:08:48.631159 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:08:48.631145    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/493338d2-24b1-47aa-a365-91e4d5274121-proxy-tls\") pod \"isvc-logger-raw-afe12-predictor-849d5f794b-6nl27\" (UID: \"493338d2-24b1-47aa-a365-91e4d5274121\") " pod="kserve-ci-e2e-test/isvc-logger-raw-afe12-predictor-849d5f794b-6nl27"
Sep 09 21:08:48.631250 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:08:48.631173    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/493338d2-24b1-47aa-a365-91e4d5274121-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-afe12-predictor-849d5f794b-6nl27\" (UID: \"493338d2-24b1-47aa-a365-91e4d5274121\") " pod="kserve-ci-e2e-test/isvc-logger-raw-afe12-predictor-849d5f794b-6nl27"
Sep 09 21:08:48.631250 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:08:48.631194    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/493338d2-24b1-47aa-a365-91e4d5274121-cabundle-cert\") pod \"isvc-logger-raw-afe12-predictor-849d5f794b-6nl27\" (UID: \"493338d2-24b1-47aa-a365-91e4d5274121\") " pod="kserve-ci-e2e-test/isvc-logger-raw-afe12-predictor-849d5f794b-6nl27"
Sep 09 21:08:48.731723 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:08:48.731698    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/493338d2-24b1-47aa-a365-91e4d5274121-kserve-provision-location\") pod \"isvc-logger-raw-afe12-predictor-849d5f794b-6nl27\" (UID: \"493338d2-24b1-47aa-a365-91e4d5274121\") " pod="kserve-ci-e2e-test/isvc-logger-raw-afe12-predictor-849d5f794b-6nl27"
Sep 09 21:08:48.731842 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:08:48.731793    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkrp5\" (UniqueName: \"kubernetes.io/projected/493338d2-24b1-47aa-a365-91e4d5274121-kube-api-access-mkrp5\") pod \"isvc-logger-raw-afe12-predictor-849d5f794b-6nl27\" (UID: \"493338d2-24b1-47aa-a365-91e4d5274121\") " pod="kserve-ci-e2e-test/isvc-logger-raw-afe12-predictor-849d5f794b-6nl27"
Sep 09 21:08:48.731842 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:08:48.731822    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/493338d2-24b1-47aa-a365-91e4d5274121-proxy-tls\") pod \"isvc-logger-raw-afe12-predictor-849d5f794b-6nl27\" (UID: \"493338d2-24b1-47aa-a365-91e4d5274121\") " pod="kserve-ci-e2e-test/isvc-logger-raw-afe12-predictor-849d5f794b-6nl27"
Sep 09 21:08:48.731935 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:08:48.731847    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/493338d2-24b1-47aa-a365-91e4d5274121-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-afe12-predictor-849d5f794b-6nl27\" (UID: \"493338d2-24b1-47aa-a365-91e4d5274121\") " pod="kserve-ci-e2e-test/isvc-logger-raw-afe12-predictor-849d5f794b-6nl27"
Sep 09 21:08:48.731935 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:08:48.731868    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/493338d2-24b1-47aa-a365-91e4d5274121-cabundle-cert\") pod \"isvc-logger-raw-afe12-predictor-849d5f794b-6nl27\" (UID: \"493338d2-24b1-47aa-a365-91e4d5274121\") " pod="kserve-ci-e2e-test/isvc-logger-raw-afe12-predictor-849d5f794b-6nl27"
Sep 09 21:08:48.732165 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:08:48.732135    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/493338d2-24b1-47aa-a365-91e4d5274121-kserve-provision-location\") pod \"isvc-logger-raw-afe12-predictor-849d5f794b-6nl27\" (UID: \"493338d2-24b1-47aa-a365-91e4d5274121\") " pod="kserve-ci-e2e-test/isvc-logger-raw-afe12-predictor-849d5f794b-6nl27"
Sep 09 21:08:48.732419 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:08:48.732399    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/493338d2-24b1-47aa-a365-91e4d5274121-cabundle-cert\") pod \"isvc-logger-raw-afe12-predictor-849d5f794b-6nl27\" (UID: \"493338d2-24b1-47aa-a365-91e4d5274121\") " pod="kserve-ci-e2e-test/isvc-logger-raw-afe12-predictor-849d5f794b-6nl27"
Sep 09 21:08:48.732521 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:08:48.732499    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/493338d2-24b1-47aa-a365-91e4d5274121-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-afe12-predictor-849d5f794b-6nl27\" (UID: \"493338d2-24b1-47aa-a365-91e4d5274121\") " pod="kserve-ci-e2e-test/isvc-logger-raw-afe12-predictor-849d5f794b-6nl27"
Sep 09 21:08:48.734450 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:08:48.734417    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/493338d2-24b1-47aa-a365-91e4d5274121-proxy-tls\") pod \"isvc-logger-raw-afe12-predictor-849d5f794b-6nl27\" (UID: \"493338d2-24b1-47aa-a365-91e4d5274121\") " pod="kserve-ci-e2e-test/isvc-logger-raw-afe12-predictor-849d5f794b-6nl27"
Sep 09 21:08:48.741165 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:08:48.741139    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkrp5\" (UniqueName: \"kubernetes.io/projected/493338d2-24b1-47aa-a365-91e4d5274121-kube-api-access-mkrp5\") pod \"isvc-logger-raw-afe12-predictor-849d5f794b-6nl27\" (UID: \"493338d2-24b1-47aa-a365-91e4d5274121\") " pod="kserve-ci-e2e-test/isvc-logger-raw-afe12-predictor-849d5f794b-6nl27"
Sep 09 21:08:48.856636 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:08:48.856579    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-afe12-predictor-849d5f794b-6nl27"
Sep 09 21:08:49.132158 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:08:49.132135    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-afe12-predictor-849d5f794b-6nl27"]
Sep 09 21:08:49.134223 ip-10-0-138-224 kubenswrapper[2579]: W0909 21:08:49.134194    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod493338d2_24b1_47aa_a365_91e4d5274121.slice/crio-64a73eee7c676947be527f83aa41aa3048d58869caacc38c603bca8e08dab80c WatchSource:0}: Error finding container 64a73eee7c676947be527f83aa41aa3048d58869caacc38c603bca8e08dab80c: Status 404 returned error can't find the container with id 64a73eee7c676947be527f83aa41aa3048d58869caacc38c603bca8e08dab80c
Sep 09 21:08:49.764926 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:08:49.764887    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-afe12-predictor-849d5f794b-6nl27" event={"ID":"493338d2-24b1-47aa-a365-91e4d5274121","Type":"ContainerStarted","Data":"3070aa591a4785816c528f158a71f01be4e90628f65f9a37e383d9ad55e76e19"}
Sep 09 21:08:49.765368 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:08:49.764969    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-afe12-predictor-849d5f794b-6nl27" event={"ID":"493338d2-24b1-47aa-a365-91e4d5274121","Type":"ContainerStarted","Data":"64a73eee7c676947be527f83aa41aa3048d58869caacc38c603bca8e08dab80c"}
Sep 09 21:08:50.630891 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:08:50.630847    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-887b3-86d9b7b5c8-fhrvh" podUID="0c7d0bde-a4ca-49d9-a084-f646b52d63f9" containerName="model-chainer-raw-hpa-887b3" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 21:08:53.784641 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:08:53.784608    2579 generic.go:356] "Generic (PLEG): container finished" podID="493338d2-24b1-47aa-a365-91e4d5274121" containerID="3070aa591a4785816c528f158a71f01be4e90628f65f9a37e383d9ad55e76e19" exitCode=0
Sep 09 21:08:53.785024 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:08:53.784681    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-afe12-predictor-849d5f794b-6nl27" event={"ID":"493338d2-24b1-47aa-a365-91e4d5274121","Type":"ContainerDied","Data":"3070aa591a4785816c528f158a71f01be4e90628f65f9a37e383d9ad55e76e19"}
Sep 09 21:08:54.792902 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:08:54.792865    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-afe12-predictor-849d5f794b-6nl27" event={"ID":"493338d2-24b1-47aa-a365-91e4d5274121","Type":"ContainerStarted","Data":"969e93e538e009544ad055167ae8f00f43d92c446069f2ab83909f2478f8c7a1"}
Sep 09 21:08:54.793275 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:08:54.792939    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-afe12-predictor-849d5f794b-6nl27" event={"ID":"493338d2-24b1-47aa-a365-91e4d5274121","Type":"ContainerStarted","Data":"e06b2fc5d20647a961e9c8ca5c812ab4f278736a6d096bf085cd8463b12b51d8"}
Sep 09 21:08:54.793275 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:08:54.792965    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-afe12-predictor-849d5f794b-6nl27" event={"ID":"493338d2-24b1-47aa-a365-91e4d5274121","Type":"ContainerStarted","Data":"c98f01b246a7f53db638e0dabaa8a181031ee40d716ec8a8c9f61cebafe370bf"}
Sep 09 21:08:54.793275 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:08:54.793242    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-afe12-predictor-849d5f794b-6nl27"
Sep 09 21:08:54.793452 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:08:54.793303    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-afe12-predictor-849d5f794b-6nl27"
Sep 09 21:08:54.793452 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:08:54.793346    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-afe12-predictor-849d5f794b-6nl27"
Sep 09 21:08:54.795029 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:08:54.794986    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-afe12-predictor-849d5f794b-6nl27" podUID="493338d2-24b1-47aa-a365-91e4d5274121" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Sep 09 21:08:54.795799 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:08:54.795777    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-afe12-predictor-849d5f794b-6nl27" podUID="493338d2-24b1-47aa-a365-91e4d5274121" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 21:08:54.817176 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:08:54.817132    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-logger-raw-afe12-predictor-849d5f794b-6nl27" podStartSLOduration=2.814010477 podStartE2EDuration="6.817118512s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.003108035s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:08:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:08:54.816794608 +0000 UTC m=+992.056888738" watchObservedRunningTime="2026-09-09 21:08:54.817118512 +0000 UTC m=+992.057212702"
Sep 09 21:08:55.631412 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:08:55.631364    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-887b3-86d9b7b5c8-fhrvh" podUID="0c7d0bde-a4ca-49d9-a084-f646b52d63f9" containerName="model-chainer-raw-hpa-887b3" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 21:08:55.798610 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:08:55.798572    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-afe12-predictor-849d5f794b-6nl27" podUID="493338d2-24b1-47aa-a365-91e4d5274121" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Sep 09 21:08:55.799040 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:08:55.799004    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-afe12-predictor-849d5f794b-6nl27" podUID="493338d2-24b1-47aa-a365-91e4d5274121" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 21:08:58.429043 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:08:58.429014    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-887b3-86d9b7b5c8-fhrvh"
Sep 09 21:08:58.505579 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:08:58.505553    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0c7d0bde-a4ca-49d9-a084-f646b52d63f9-proxy-tls\") pod \"0c7d0bde-a4ca-49d9-a084-f646b52d63f9\" (UID: \"0c7d0bde-a4ca-49d9-a084-f646b52d63f9\") "
Sep 09 21:08:58.505759 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:08:58.505596    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0c7d0bde-a4ca-49d9-a084-f646b52d63f9-openshift-service-ca-bundle\") pod \"0c7d0bde-a4ca-49d9-a084-f646b52d63f9\" (UID: \"0c7d0bde-a4ca-49d9-a084-f646b52d63f9\") "
Sep 09 21:08:58.506006 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:08:58.505973    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c7d0bde-a4ca-49d9-a084-f646b52d63f9-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "0c7d0bde-a4ca-49d9-a084-f646b52d63f9" (UID: "0c7d0bde-a4ca-49d9-a084-f646b52d63f9"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:08:58.507785 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:08:58.507763    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c7d0bde-a4ca-49d9-a084-f646b52d63f9-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "0c7d0bde-a4ca-49d9-a084-f646b52d63f9" (UID: "0c7d0bde-a4ca-49d9-a084-f646b52d63f9"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:08:58.606254 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:08:58.606179    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0c7d0bde-a4ca-49d9-a084-f646b52d63f9-proxy-tls\") on node \"ip-10-0-138-224.ec2.internal\" DevicePath \"\""
Sep 09 21:08:58.606254 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:08:58.606207    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0c7d0bde-a4ca-49d9-a084-f646b52d63f9-openshift-service-ca-bundle\") on node \"ip-10-0-138-224.ec2.internal\" DevicePath \"\""
Sep 09 21:08:58.812196 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:08:58.812157    2579 generic.go:356] "Generic (PLEG): container finished" podID="0c7d0bde-a4ca-49d9-a084-f646b52d63f9" containerID="39113329548a81dce645d9b8d1d7b3e0b49f9cd33500c034f98b3d458e8359a9" exitCode=0
Sep 09 21:08:58.812196 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:08:58.812194    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-887b3-86d9b7b5c8-fhrvh"
Sep 09 21:08:58.812429 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:08:58.812226    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-887b3-86d9b7b5c8-fhrvh" event={"ID":"0c7d0bde-a4ca-49d9-a084-f646b52d63f9","Type":"ContainerDied","Data":"39113329548a81dce645d9b8d1d7b3e0b49f9cd33500c034f98b3d458e8359a9"}
Sep 09 21:08:58.812429 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:08:58.812285    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-887b3-86d9b7b5c8-fhrvh" event={"ID":"0c7d0bde-a4ca-49d9-a084-f646b52d63f9","Type":"ContainerDied","Data":"7e010ea4a12309cd43d7fb02d080372e831a28a8c343e6c0021942bba46b84c5"}
Sep 09 21:08:58.812429 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:08:58.812315    2579 scope.go:117] "RemoveContainer" containerID="39113329548a81dce645d9b8d1d7b3e0b49f9cd33500c034f98b3d458e8359a9"
Sep 09 21:08:58.820512 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:08:58.820495    2579 scope.go:117] "RemoveContainer" containerID="39113329548a81dce645d9b8d1d7b3e0b49f9cd33500c034f98b3d458e8359a9"
Sep 09 21:08:58.820778 ip-10-0-138-224 kubenswrapper[2579]: E0909 21:08:58.820760    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39113329548a81dce645d9b8d1d7b3e0b49f9cd33500c034f98b3d458e8359a9\": container with ID starting with 39113329548a81dce645d9b8d1d7b3e0b49f9cd33500c034f98b3d458e8359a9 not found: ID does not exist" containerID="39113329548a81dce645d9b8d1d7b3e0b49f9cd33500c034f98b3d458e8359a9"
Sep 09 21:08:58.820838 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:08:58.820789    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39113329548a81dce645d9b8d1d7b3e0b49f9cd33500c034f98b3d458e8359a9"} err="failed to get container status \"39113329548a81dce645d9b8d1d7b3e0b49f9cd33500c034f98b3d458e8359a9\": rpc error: code = NotFound desc = could not find container \"39113329548a81dce645d9b8d1d7b3e0b49f9cd33500c034f98b3d458e8359a9\": container with ID starting with 39113329548a81dce645d9b8d1d7b3e0b49f9cd33500c034f98b3d458e8359a9 not found: ID does not exist"
Sep 09 21:08:58.833937 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:08:58.833908    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-hpa-887b3-86d9b7b5c8-fhrvh"]
Sep 09 21:08:58.837663 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:08:58.837644    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-hpa-887b3-86d9b7b5c8-fhrvh"]
Sep 09 21:08:59.291114 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:08:59.291081    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0c7d0bde-a4ca-49d9-a084-f646b52d63f9" path="/var/lib/kubelet/pods/0c7d0bde-a4ca-49d9-a084-f646b52d63f9/volumes"
Sep 09 21:09:00.803236 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:09:00.803211    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-afe12-predictor-849d5f794b-6nl27"
Sep 09 21:09:00.803926 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:09:00.803892    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-afe12-predictor-849d5f794b-6nl27" podUID="493338d2-24b1-47aa-a365-91e4d5274121" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Sep 09 21:09:00.804237 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:09:00.804202    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-afe12-predictor-849d5f794b-6nl27" podUID="493338d2-24b1-47aa-a365-91e4d5274121" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 21:09:10.804762 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:09:10.804713    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-afe12-predictor-849d5f794b-6nl27" podUID="493338d2-24b1-47aa-a365-91e4d5274121" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Sep 09 21:09:10.805213 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:09:10.805137    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-afe12-predictor-849d5f794b-6nl27" podUID="493338d2-24b1-47aa-a365-91e4d5274121" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 21:09:20.804346 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:09:20.804303    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-afe12-predictor-849d5f794b-6nl27" podUID="493338d2-24b1-47aa-a365-91e4d5274121" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Sep 09 21:09:20.804914 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:09:20.804764    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-afe12-predictor-849d5f794b-6nl27" podUID="493338d2-24b1-47aa-a365-91e4d5274121" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 21:09:30.804372 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:09:30.804318    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-afe12-predictor-849d5f794b-6nl27" podUID="493338d2-24b1-47aa-a365-91e4d5274121" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Sep 09 21:09:30.804798 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:09:30.804761    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-afe12-predictor-849d5f794b-6nl27" podUID="493338d2-24b1-47aa-a365-91e4d5274121" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 21:09:40.804228 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:09:40.804180    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-afe12-predictor-849d5f794b-6nl27" podUID="493338d2-24b1-47aa-a365-91e4d5274121" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Sep 09 21:09:40.804657 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:09:40.804618    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-afe12-predictor-849d5f794b-6nl27" podUID="493338d2-24b1-47aa-a365-91e4d5274121" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 21:09:43.390632 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:09:43.390599    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-ea23b7-predictor-5678c7c4cb-k8592"]
Sep 09 21:09:43.391126 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:09:43.391108    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0c7d0bde-a4ca-49d9-a084-f646b52d63f9" containerName="model-chainer-raw-hpa-887b3"
Sep 09 21:09:43.391185 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:09:43.391130    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="0c7d0bde-a4ca-49d9-a084-f646b52d63f9" containerName="model-chainer-raw-hpa-887b3"
Sep 09 21:09:43.391226 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:09:43.391196    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="0c7d0bde-a4ca-49d9-a084-f646b52d63f9" containerName="model-chainer-raw-hpa-887b3"
Sep 09 21:09:43.395822 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:09:43.395800    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-ea23b7-predictor-5678c7c4cb-k8592"
Sep 09 21:09:43.398334 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:09:43.398312    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-secret-ea23b7\""
Sep 09 21:09:43.398682 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:09:43.398662    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-ea23b7-predictor-serving-cert\""
Sep 09 21:09:43.398788 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:09:43.398686    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-ea23b7-kube-rbac-proxy-sar-config\""
Sep 09 21:09:43.398788 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:09:43.398710    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-sa-ea23b7-dockercfg-rjkrt\""
Sep 09 21:09:43.404545 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:09:43.404522    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-ea23b7-predictor-5678c7c4cb-k8592"]
Sep 09 21:09:43.466915 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:09:43.466891    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b1d7eb00-62da-4862-b21f-035980cbf2bc-cabundle-cert\") pod \"isvc-secondary-ea23b7-predictor-5678c7c4cb-k8592\" (UID: \"b1d7eb00-62da-4862-b21f-035980cbf2bc\") " pod="kserve-ci-e2e-test/isvc-secondary-ea23b7-predictor-5678c7c4cb-k8592"
Sep 09 21:09:43.467022 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:09:43.466934    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b1d7eb00-62da-4862-b21f-035980cbf2bc-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-ea23b7-predictor-5678c7c4cb-k8592\" (UID: \"b1d7eb00-62da-4862-b21f-035980cbf2bc\") " pod="kserve-ci-e2e-test/isvc-secondary-ea23b7-predictor-5678c7c4cb-k8592"
Sep 09 21:09:43.467022 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:09:43.467009    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b1d7eb00-62da-4862-b21f-035980cbf2bc-proxy-tls\") pod \"isvc-secondary-ea23b7-predictor-5678c7c4cb-k8592\" (UID: \"b1d7eb00-62da-4862-b21f-035980cbf2bc\") " pod="kserve-ci-e2e-test/isvc-secondary-ea23b7-predictor-5678c7c4cb-k8592"
Sep 09 21:09:43.467108 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:09:43.467057    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b1d7eb00-62da-4862-b21f-035980cbf2bc-kserve-provision-location\") pod \"isvc-secondary-ea23b7-predictor-5678c7c4cb-k8592\" (UID: \"b1d7eb00-62da-4862-b21f-035980cbf2bc\") " pod="kserve-ci-e2e-test/isvc-secondary-ea23b7-predictor-5678c7c4cb-k8592"
Sep 09 21:09:43.467149 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:09:43.467112    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xcrhf\" (UniqueName: \"kubernetes.io/projected/b1d7eb00-62da-4862-b21f-035980cbf2bc-kube-api-access-xcrhf\") pod \"isvc-secondary-ea23b7-predictor-5678c7c4cb-k8592\" (UID: \"b1d7eb00-62da-4862-b21f-035980cbf2bc\") " pod="kserve-ci-e2e-test/isvc-secondary-ea23b7-predictor-5678c7c4cb-k8592"
Sep 09 21:09:43.567790 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:09:43.567737    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b1d7eb00-62da-4862-b21f-035980cbf2bc-kserve-provision-location\") pod \"isvc-secondary-ea23b7-predictor-5678c7c4cb-k8592\" (UID: \"b1d7eb00-62da-4862-b21f-035980cbf2bc\") " pod="kserve-ci-e2e-test/isvc-secondary-ea23b7-predictor-5678c7c4cb-k8592"
Sep 09 21:09:43.567940 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:09:43.567804    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xcrhf\" (UniqueName: \"kubernetes.io/projected/b1d7eb00-62da-4862-b21f-035980cbf2bc-kube-api-access-xcrhf\") pod \"isvc-secondary-ea23b7-predictor-5678c7c4cb-k8592\" (UID: \"b1d7eb00-62da-4862-b21f-035980cbf2bc\") " pod="kserve-ci-e2e-test/isvc-secondary-ea23b7-predictor-5678c7c4cb-k8592"
Sep 09 21:09:43.567940 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:09:43.567838    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b1d7eb00-62da-4862-b21f-035980cbf2bc-cabundle-cert\") pod \"isvc-secondary-ea23b7-predictor-5678c7c4cb-k8592\" (UID: \"b1d7eb00-62da-4862-b21f-035980cbf2bc\") " pod="kserve-ci-e2e-test/isvc-secondary-ea23b7-predictor-5678c7c4cb-k8592"
Sep 09 21:09:43.567940 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:09:43.567869    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b1d7eb00-62da-4862-b21f-035980cbf2bc-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-ea23b7-predictor-5678c7c4cb-k8592\" (UID: \"b1d7eb00-62da-4862-b21f-035980cbf2bc\") " pod="kserve-ci-e2e-test/isvc-secondary-ea23b7-predictor-5678c7c4cb-k8592"
Sep 09 21:09:43.567940 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:09:43.567904    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b1d7eb00-62da-4862-b21f-035980cbf2bc-proxy-tls\") pod \"isvc-secondary-ea23b7-predictor-5678c7c4cb-k8592\" (UID: \"b1d7eb00-62da-4862-b21f-035980cbf2bc\") " pod="kserve-ci-e2e-test/isvc-secondary-ea23b7-predictor-5678c7c4cb-k8592"
Sep 09 21:09:43.568219 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:09:43.568198    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b1d7eb00-62da-4862-b21f-035980cbf2bc-kserve-provision-location\") pod \"isvc-secondary-ea23b7-predictor-5678c7c4cb-k8592\" (UID: \"b1d7eb00-62da-4862-b21f-035980cbf2bc\") " pod="kserve-ci-e2e-test/isvc-secondary-ea23b7-predictor-5678c7c4cb-k8592"
Sep 09 21:09:43.568567 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:09:43.568546    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b1d7eb00-62da-4862-b21f-035980cbf2bc-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-ea23b7-predictor-5678c7c4cb-k8592\" (UID: \"b1d7eb00-62da-4862-b21f-035980cbf2bc\") " pod="kserve-ci-e2e-test/isvc-secondary-ea23b7-predictor-5678c7c4cb-k8592"
Sep 09 21:09:43.568647 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:09:43.568584    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b1d7eb00-62da-4862-b21f-035980cbf2bc-cabundle-cert\") pod \"isvc-secondary-ea23b7-predictor-5678c7c4cb-k8592\" (UID: \"b1d7eb00-62da-4862-b21f-035980cbf2bc\") " pod="kserve-ci-e2e-test/isvc-secondary-ea23b7-predictor-5678c7c4cb-k8592"
Sep 09 21:09:43.570492 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:09:43.570476    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b1d7eb00-62da-4862-b21f-035980cbf2bc-proxy-tls\") pod \"isvc-secondary-ea23b7-predictor-5678c7c4cb-k8592\" (UID: \"b1d7eb00-62da-4862-b21f-035980cbf2bc\") " pod="kserve-ci-e2e-test/isvc-secondary-ea23b7-predictor-5678c7c4cb-k8592"
Sep 09 21:09:43.577318 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:09:43.577298    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xcrhf\" (UniqueName: \"kubernetes.io/projected/b1d7eb00-62da-4862-b21f-035980cbf2bc-kube-api-access-xcrhf\") pod \"isvc-secondary-ea23b7-predictor-5678c7c4cb-k8592\" (UID: \"b1d7eb00-62da-4862-b21f-035980cbf2bc\") " pod="kserve-ci-e2e-test/isvc-secondary-ea23b7-predictor-5678c7c4cb-k8592"
Sep 09 21:09:43.706535 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:09:43.706512    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-ea23b7-predictor-5678c7c4cb-k8592"
Sep 09 21:09:43.980420 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:09:43.980379    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-ea23b7-predictor-5678c7c4cb-k8592"]
Sep 09 21:09:43.983100 ip-10-0-138-224 kubenswrapper[2579]: W0909 21:09:43.983057    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1d7eb00_62da_4862_b21f_035980cbf2bc.slice/crio-7ce31ab600c631ac7a799f7c0c00ce31ff582597f03fdffe44a5f210eb70169e WatchSource:0}: Error finding container 7ce31ab600c631ac7a799f7c0c00ce31ff582597f03fdffe44a5f210eb70169e: Status 404 returned error can't find the container with id 7ce31ab600c631ac7a799f7c0c00ce31ff582597f03fdffe44a5f210eb70169e
Sep 09 21:09:44.016990 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:09:44.016963    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-ea23b7-predictor-5678c7c4cb-k8592" event={"ID":"b1d7eb00-62da-4862-b21f-035980cbf2bc","Type":"ContainerStarted","Data":"7ce31ab600c631ac7a799f7c0c00ce31ff582597f03fdffe44a5f210eb70169e"}
Sep 09 21:09:45.029036 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:09:45.029002    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-ea23b7-predictor-5678c7c4cb-k8592" event={"ID":"b1d7eb00-62da-4862-b21f-035980cbf2bc","Type":"ContainerStarted","Data":"fa1ab1501584e9f0c7f5c9e6fdb17d4064a1298af636b73f910d0ddfca8540c5"}
Sep 09 21:09:50.804090 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:09:50.804049    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-afe12-predictor-849d5f794b-6nl27" podUID="493338d2-24b1-47aa-a365-91e4d5274121" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Sep 09 21:09:50.804607 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:09:50.804584    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-afe12-predictor-849d5f794b-6nl27" podUID="493338d2-24b1-47aa-a365-91e4d5274121" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 21:09:51.055963 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:09:51.055895    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-ea23b7-predictor-5678c7c4cb-k8592_b1d7eb00-62da-4862-b21f-035980cbf2bc/storage-initializer/0.log"
Sep 09 21:09:51.055963 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:09:51.055930    2579 generic.go:356] "Generic (PLEG): container finished" podID="b1d7eb00-62da-4862-b21f-035980cbf2bc" containerID="fa1ab1501584e9f0c7f5c9e6fdb17d4064a1298af636b73f910d0ddfca8540c5" exitCode=1
Sep 09 21:09:51.056150 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:09:51.056010    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-ea23b7-predictor-5678c7c4cb-k8592" event={"ID":"b1d7eb00-62da-4862-b21f-035980cbf2bc","Type":"ContainerDied","Data":"fa1ab1501584e9f0c7f5c9e6fdb17d4064a1298af636b73f910d0ddfca8540c5"}
Sep 09 21:09:52.062203 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:09:52.062173    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-ea23b7-predictor-5678c7c4cb-k8592_b1d7eb00-62da-4862-b21f-035980cbf2bc/storage-initializer/0.log"
Sep 09 21:09:52.062652 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:09:52.062222    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-ea23b7-predictor-5678c7c4cb-k8592" event={"ID":"b1d7eb00-62da-4862-b21f-035980cbf2bc","Type":"ContainerStarted","Data":"337c4ba43fcb1146d39571dbab05f93c2ff9a491f5020ba85042950fcc5721fc"}
Sep 09 21:09:57.087059 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:09:57.087030    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-ea23b7-predictor-5678c7c4cb-k8592_b1d7eb00-62da-4862-b21f-035980cbf2bc/storage-initializer/1.log"
Sep 09 21:09:57.087481 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:09:57.087465    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-ea23b7-predictor-5678c7c4cb-k8592_b1d7eb00-62da-4862-b21f-035980cbf2bc/storage-initializer/0.log"
Sep 09 21:09:57.087540 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:09:57.087500    2579 generic.go:356] "Generic (PLEG): container finished" podID="b1d7eb00-62da-4862-b21f-035980cbf2bc" containerID="337c4ba43fcb1146d39571dbab05f93c2ff9a491f5020ba85042950fcc5721fc" exitCode=1
Sep 09 21:09:57.087592 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:09:57.087574    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-ea23b7-predictor-5678c7c4cb-k8592" event={"ID":"b1d7eb00-62da-4862-b21f-035980cbf2bc","Type":"ContainerDied","Data":"337c4ba43fcb1146d39571dbab05f93c2ff9a491f5020ba85042950fcc5721fc"}
Sep 09 21:09:57.087640 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:09:57.087624    2579 scope.go:117] "RemoveContainer" containerID="fa1ab1501584e9f0c7f5c9e6fdb17d4064a1298af636b73f910d0ddfca8540c5"
Sep 09 21:09:57.088009 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:09:57.087980    2579 scope.go:117] "RemoveContainer" containerID="fa1ab1501584e9f0c7f5c9e6fdb17d4064a1298af636b73f910d0ddfca8540c5"
Sep 09 21:09:57.102666 ip-10-0-138-224 kubenswrapper[2579]: E0909 21:09:57.102637    2579 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-ea23b7-predictor-5678c7c4cb-k8592_kserve-ci-e2e-test_b1d7eb00-62da-4862-b21f-035980cbf2bc_0 in pod sandbox 7ce31ab600c631ac7a799f7c0c00ce31ff582597f03fdffe44a5f210eb70169e from index: no such id: 'fa1ab1501584e9f0c7f5c9e6fdb17d4064a1298af636b73f910d0ddfca8540c5'" containerID="fa1ab1501584e9f0c7f5c9e6fdb17d4064a1298af636b73f910d0ddfca8540c5"
Sep 09 21:09:57.102728 ip-10-0-138-224 kubenswrapper[2579]: E0909 21:09:57.102684    2579 kuberuntime_container.go:962] "Unhandled Error" err="failed to remove pod init container \"storage-initializer\": rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-ea23b7-predictor-5678c7c4cb-k8592_kserve-ci-e2e-test_b1d7eb00-62da-4862-b21f-035980cbf2bc_0 in pod sandbox 7ce31ab600c631ac7a799f7c0c00ce31ff582597f03fdffe44a5f210eb70169e from index: no such id: 'fa1ab1501584e9f0c7f5c9e6fdb17d4064a1298af636b73f910d0ddfca8540c5'; Skipping pod \"isvc-secondary-ea23b7-predictor-5678c7c4cb-k8592_kserve-ci-e2e-test(b1d7eb00-62da-4862-b21f-035980cbf2bc)\"" logger="UnhandledError"
Sep 09 21:09:57.102881 ip-10-0-138-224 kubenswrapper[2579]: E0909 21:09:57.102864    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-secondary-ea23b7-predictor-5678c7c4cb-k8592_kserve-ci-e2e-test(b1d7eb00-62da-4862-b21f-035980cbf2bc)\"" pod="kserve-ci-e2e-test/isvc-secondary-ea23b7-predictor-5678c7c4cb-k8592" podUID="b1d7eb00-62da-4862-b21f-035980cbf2bc"
Sep 09 21:09:58.093139 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:09:58.093112    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-ea23b7-predictor-5678c7c4cb-k8592_b1d7eb00-62da-4862-b21f-035980cbf2bc/storage-initializer/1.log"
Sep 09 21:10:00.803957 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:10:00.803882    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-afe12-predictor-849d5f794b-6nl27" podUID="493338d2-24b1-47aa-a365-91e4d5274121" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Sep 09 21:10:00.804465 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:10:00.804284    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-afe12-predictor-849d5f794b-6nl27" podUID="493338d2-24b1-47aa-a365-91e4d5274121" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 21:10:01.445780 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:10:01.445748    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-ea23b7-predictor-5678c7c4cb-k8592"]
Sep 09 21:10:01.579961 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:10:01.579941    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-ea23b7-predictor-5678c7c4cb-k8592_b1d7eb00-62da-4862-b21f-035980cbf2bc/storage-initializer/1.log"
Sep 09 21:10:01.583669 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:10:01.583649    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-ea23b7-predictor-5678c7c4cb-k8592"
Sep 09 21:10:01.697499 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:10:01.697419    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b1d7eb00-62da-4862-b21f-035980cbf2bc-cabundle-cert\") pod \"b1d7eb00-62da-4862-b21f-035980cbf2bc\" (UID: \"b1d7eb00-62da-4862-b21f-035980cbf2bc\") "
Sep 09 21:10:01.697499 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:10:01.697475    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xcrhf\" (UniqueName: \"kubernetes.io/projected/b1d7eb00-62da-4862-b21f-035980cbf2bc-kube-api-access-xcrhf\") pod \"b1d7eb00-62da-4862-b21f-035980cbf2bc\" (UID: \"b1d7eb00-62da-4862-b21f-035980cbf2bc\") "
Sep 09 21:10:01.697687 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:10:01.697516    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b1d7eb00-62da-4862-b21f-035980cbf2bc-kube-rbac-proxy-sar-config\") pod \"b1d7eb00-62da-4862-b21f-035980cbf2bc\" (UID: \"b1d7eb00-62da-4862-b21f-035980cbf2bc\") "
Sep 09 21:10:01.697687 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:10:01.697564    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b1d7eb00-62da-4862-b21f-035980cbf2bc-proxy-tls\") pod \"b1d7eb00-62da-4862-b21f-035980cbf2bc\" (UID: \"b1d7eb00-62da-4862-b21f-035980cbf2bc\") "
Sep 09 21:10:01.697687 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:10:01.697588    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b1d7eb00-62da-4862-b21f-035980cbf2bc-kserve-provision-location\") pod \"b1d7eb00-62da-4862-b21f-035980cbf2bc\" (UID: \"b1d7eb00-62da-4862-b21f-035980cbf2bc\") "
Sep 09 21:10:01.697850 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:10:01.697798    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1d7eb00-62da-4862-b21f-035980cbf2bc-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "b1d7eb00-62da-4862-b21f-035980cbf2bc" (UID: "b1d7eb00-62da-4862-b21f-035980cbf2bc"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:10:01.697904 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:10:01.697843    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1d7eb00-62da-4862-b21f-035980cbf2bc-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "b1d7eb00-62da-4862-b21f-035980cbf2bc" (UID: "b1d7eb00-62da-4862-b21f-035980cbf2bc"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:10:01.697957 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:10:01.697917    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1d7eb00-62da-4862-b21f-035980cbf2bc-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b1d7eb00-62da-4862-b21f-035980cbf2bc" (UID: "b1d7eb00-62da-4862-b21f-035980cbf2bc"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:10:01.699699 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:10:01.699675    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1d7eb00-62da-4862-b21f-035980cbf2bc-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b1d7eb00-62da-4862-b21f-035980cbf2bc" (UID: "b1d7eb00-62da-4862-b21f-035980cbf2bc"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:10:01.699827 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:10:01.699725    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1d7eb00-62da-4862-b21f-035980cbf2bc-kube-api-access-xcrhf" (OuterVolumeSpecName: "kube-api-access-xcrhf") pod "b1d7eb00-62da-4862-b21f-035980cbf2bc" (UID: "b1d7eb00-62da-4862-b21f-035980cbf2bc"). InnerVolumeSpecName "kube-api-access-xcrhf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:10:01.799081 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:10:01.799051    2579 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b1d7eb00-62da-4862-b21f-035980cbf2bc-cabundle-cert\") on node \"ip-10-0-138-224.ec2.internal\" DevicePath \"\""
Sep 09 21:10:01.799081 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:10:01.799078    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xcrhf\" (UniqueName: \"kubernetes.io/projected/b1d7eb00-62da-4862-b21f-035980cbf2bc-kube-api-access-xcrhf\") on node \"ip-10-0-138-224.ec2.internal\" DevicePath \"\""
Sep 09 21:10:01.799245 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:10:01.799092    2579 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b1d7eb00-62da-4862-b21f-035980cbf2bc-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-224.ec2.internal\" DevicePath \"\""
Sep 09 21:10:01.799245 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:10:01.799104    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b1d7eb00-62da-4862-b21f-035980cbf2bc-proxy-tls\") on node \"ip-10-0-138-224.ec2.internal\" DevicePath \"\""
Sep 09 21:10:01.799245 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:10:01.799118    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b1d7eb00-62da-4862-b21f-035980cbf2bc-kserve-provision-location\") on node \"ip-10-0-138-224.ec2.internal\" DevicePath \"\""
Sep 09 21:10:02.112706 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:10:02.112621    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-ea23b7-predictor-5678c7c4cb-k8592_b1d7eb00-62da-4862-b21f-035980cbf2bc/storage-initializer/1.log"
Sep 09 21:10:02.113111 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:10:02.112714    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-ea23b7-predictor-5678c7c4cb-k8592" event={"ID":"b1d7eb00-62da-4862-b21f-035980cbf2bc","Type":"ContainerDied","Data":"7ce31ab600c631ac7a799f7c0c00ce31ff582597f03fdffe44a5f210eb70169e"}
Sep 09 21:10:02.113111 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:10:02.112782    2579 scope.go:117] "RemoveContainer" containerID="337c4ba43fcb1146d39571dbab05f93c2ff9a491f5020ba85042950fcc5721fc"
Sep 09 21:10:02.113111 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:10:02.112723    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-ea23b7-predictor-5678c7c4cb-k8592"
Sep 09 21:10:02.146847 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:10:02.146821    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-ea23b7-predictor-5678c7c4cb-k8592"]
Sep 09 21:10:02.153513 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:10:02.153485    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-ea23b7-predictor-5678c7c4cb-k8592"]
Sep 09 21:10:03.292020 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:10:03.291985    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b1d7eb00-62da-4862-b21f-035980cbf2bc" path="/var/lib/kubelet/pods/b1d7eb00-62da-4862-b21f-035980cbf2bc/volumes"
Sep 09 21:10:10.804702 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:10:10.804669    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-afe12-predictor-849d5f794b-6nl27"
Sep 09 21:10:10.805098 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:10:10.804947    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-afe12-predictor-849d5f794b-6nl27"
Sep 09 21:10:23.535919 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:10:23.535897    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_message-dumper-raw-afe12-predictor-57759dbb49-s4j9t_e722d823-93a6-4495-9e58-0407ab0d8fd5/kserve-container/0.log"
Sep 09 21:10:23.694401 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:10:23.694359    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-afe12-predictor-849d5f794b-6nl27"]
Sep 09 21:10:23.694687 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:10:23.694661    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-afe12-predictor-849d5f794b-6nl27" podUID="493338d2-24b1-47aa-a365-91e4d5274121" containerName="agent" containerID="cri-o://969e93e538e009544ad055167ae8f00f43d92c446069f2ab83909f2478f8c7a1" gracePeriod=30
Sep 09 21:10:23.694792 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:10:23.694644    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-afe12-predictor-849d5f794b-6nl27" podUID="493338d2-24b1-47aa-a365-91e4d5274121" containerName="kserve-container" containerID="cri-o://c98f01b246a7f53db638e0dabaa8a181031ee40d716ec8a8c9f61cebafe370bf" gracePeriod=30
Sep 09 21:10:23.694792 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:10:23.694735    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-afe12-predictor-849d5f794b-6nl27" podUID="493338d2-24b1-47aa-a365-91e4d5274121" containerName="kube-rbac-proxy" containerID="cri-o://e06b2fc5d20647a961e9c8ca5c812ab4f278736a6d096bf085cd8463b12b51d8" gracePeriod=30
Sep 09 21:10:23.756880 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:10:23.756849    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-afe12-predictor-57759dbb49-s4j9t"]
Sep 09 21:10:23.757120 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:10:23.757077    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-raw-afe12-predictor-57759dbb49-s4j9t" podUID="e722d823-93a6-4495-9e58-0407ab0d8fd5" containerName="kserve-container" containerID="cri-o://91f6c681920fd47dc1ad558b8355ec371b6cf5c0bcb7d7dfad4188eb8f7de5ef" gracePeriod=30
Sep 09 21:10:23.757188 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:10:23.757143    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-raw-afe12-predictor-57759dbb49-s4j9t" podUID="e722d823-93a6-4495-9e58-0407ab0d8fd5" containerName="kube-rbac-proxy" containerID="cri-o://cab9bc58008dcaedde5054741521a50b2d36a09e5f5ec653e16318451c05d771" gracePeriod=30
Sep 09 21:10:23.969964 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:10:23.969884    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-762cd-predictor-6b8d96d748-lw8dj"]
Sep 09 21:10:23.970467 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:10:23.970430    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1d7eb00-62da-4862-b21f-035980cbf2bc" containerName="storage-initializer"
Sep 09 21:10:23.970467 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:10:23.970469    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1d7eb00-62da-4862-b21f-035980cbf2bc" containerName="storage-initializer"
Sep 09 21:10:23.970631 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:10:23.970483    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1d7eb00-62da-4862-b21f-035980cbf2bc" containerName="storage-initializer"
Sep 09 21:10:23.970631 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:10:23.970493    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1d7eb00-62da-4862-b21f-035980cbf2bc" containerName="storage-initializer"
Sep 09 21:10:23.970631 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:10:23.970601    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="b1d7eb00-62da-4862-b21f-035980cbf2bc" containerName="storage-initializer"
Sep 09 21:10:23.970631 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:10:23.970614    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="b1d7eb00-62da-4862-b21f-035980cbf2bc" containerName="storage-initializer"
Sep 09 21:10:23.976582 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:10:23.976554    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-762cd-predictor-6b8d96d748-lw8dj"
Sep 09 21:10:23.980164 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:10:23.980137    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-scale-raw-762cd-kube-rbac-proxy-sar-config\""
Sep 09 21:10:23.980270 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:10:23.980142    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-scale-raw-762cd-predictor-serving-cert\""
Sep 09 21:10:23.986767 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:10:23.986741    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-762cd-predictor-6b8d96d748-lw8dj"]
Sep 09 21:10:24.015748 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:10:24.015727    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-afe12-predictor-57759dbb49-s4j9t"
Sep 09 21:10:24.080497 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:10:24.080463    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e722d823-93a6-4495-9e58-0407ab0d8fd5-proxy-tls\") pod \"e722d823-93a6-4495-9e58-0407ab0d8fd5\" (UID: \"e722d823-93a6-4495-9e58-0407ab0d8fd5\") "
Sep 09 21:10:24.080650 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:10:24.080549    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e722d823-93a6-4495-9e58-0407ab0d8fd5-kube-rbac-proxy-sar-config\") pod \"e722d823-93a6-4495-9e58-0407ab0d8fd5\" (UID: \"e722d823-93a6-4495-9e58-0407ab0d8fd5\") "
Sep 09 21:10:24.080650 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:10:24.080585    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2r7pm\" (UniqueName: \"kubernetes.io/projected/e722d823-93a6-4495-9e58-0407ab0d8fd5-kube-api-access-2r7pm\") pod \"e722d823-93a6-4495-9e58-0407ab0d8fd5\" (UID: \"e722d823-93a6-4495-9e58-0407ab0d8fd5\") "
Sep 09 21:10:24.080764 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:10:24.080669    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79w4j\" (UniqueName: \"kubernetes.io/projected/d0c5c989-6102-415f-9bfc-faeb8eec7311-kube-api-access-79w4j\") pod \"isvc-sklearn-scale-raw-762cd-predictor-6b8d96d748-lw8dj\" (UID: \"d0c5c989-6102-415f-9bfc-faeb8eec7311\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-762cd-predictor-6b8d96d748-lw8dj"
Sep 09 21:10:24.080764 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:10:24.080694    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d0c5c989-6102-415f-9bfc-faeb8eec7311-proxy-tls\") pod \"isvc-sklearn-scale-raw-762cd-predictor-6b8d96d748-lw8dj\" (UID: \"d0c5c989-6102-415f-9bfc-faeb8eec7311\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-762cd-predictor-6b8d96d748-lw8dj"
Sep 09 21:10:24.080764 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:10:24.080731    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d0c5c989-6102-415f-9bfc-faeb8eec7311-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-762cd-predictor-6b8d96d748-lw8dj\" (UID: \"d0c5c989-6102-415f-9bfc-faeb8eec7311\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-762cd-predictor-6b8d96d748-lw8dj"
Sep 09 21:10:24.080924 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:10:24.080783    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d0c5c989-6102-415f-9bfc-faeb8eec7311-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-762cd-predictor-6b8d96d748-lw8dj\" (UID: \"d0c5c989-6102-415f-9bfc-faeb8eec7311\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-762cd-predictor-6b8d96d748-lw8dj"
Sep 09 21:10:24.080924 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:10:24.080879    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d0c5c989-6102-415f-9bfc-faeb8eec7311-cabundle-cert\") pod \"isvc-sklearn-scale-raw-762cd-predictor-6b8d96d748-lw8dj\" (UID: \"d0c5c989-6102-415f-9bfc-faeb8eec7311\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-762cd-predictor-6b8d96d748-lw8dj"
Sep 09 21:10:24.081026 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:10:24.080930    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e722d823-93a6-4495-9e58-0407ab0d8fd5-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e722d823-93a6-4495-9e58-0407ab0d8fd5" (UID: "e722d823-93a6-4495-9e58-0407ab0d8fd5"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:10:24.082673 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:10:24.082646    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e722d823-93a6-4495-9e58-0407ab0d8fd5-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e722d823-93a6-4495-9e58-0407ab0d8fd5" (UID: "e722d823-93a6-4495-9e58-0407ab0d8fd5"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:10:24.082775 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:10:24.082742    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e722d823-93a6-4495-9e58-0407ab0d8fd5-kube-api-access-2r7pm" (OuterVolumeSpecName: "kube-api-access-2r7pm") pod "e722d823-93a6-4495-9e58-0407ab0d8fd5" (UID: "e722d823-93a6-4495-9e58-0407ab0d8fd5"). InnerVolumeSpecName "kube-api-access-2r7pm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:10:24.182120 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:10:24.182084    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d0c5c989-6102-415f-9bfc-faeb8eec7311-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-762cd-predictor-6b8d96d748-lw8dj\" (UID: \"d0c5c989-6102-415f-9bfc-faeb8eec7311\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-762cd-predictor-6b8d96d748-lw8dj"
Sep 09 21:10:24.182283 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:10:24.182138    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d0c5c989-6102-415f-9bfc-faeb8eec7311-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-762cd-predictor-6b8d96d748-lw8dj\" (UID: \"d0c5c989-6102-415f-9bfc-faeb8eec7311\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-762cd-predictor-6b8d96d748-lw8dj"
Sep 09 21:10:24.182283 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:10:24.182190    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d0c5c989-6102-415f-9bfc-faeb8eec7311-cabundle-cert\") pod \"isvc-sklearn-scale-raw-762cd-predictor-6b8d96d748-lw8dj\" (UID: \"d0c5c989-6102-415f-9bfc-faeb8eec7311\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-762cd-predictor-6b8d96d748-lw8dj"
Sep 09 21:10:24.182283 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:10:24.182238    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79w4j\" (UniqueName: \"kubernetes.io/projected/d0c5c989-6102-415f-9bfc-faeb8eec7311-kube-api-access-79w4j\") pod \"isvc-sklearn-scale-raw-762cd-predictor-6b8d96d748-lw8dj\" (UID: \"d0c5c989-6102-415f-9bfc-faeb8eec7311\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-762cd-predictor-6b8d96d748-lw8dj"
Sep 09 21:10:24.182283 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:10:24.182271    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d0c5c989-6102-415f-9bfc-faeb8eec7311-proxy-tls\") pod \"isvc-sklearn-scale-raw-762cd-predictor-6b8d96d748-lw8dj\" (UID: \"d0c5c989-6102-415f-9bfc-faeb8eec7311\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-762cd-predictor-6b8d96d748-lw8dj"
Sep 09 21:10:24.182544 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:10:24.182312    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e722d823-93a6-4495-9e58-0407ab0d8fd5-proxy-tls\") on node \"ip-10-0-138-224.ec2.internal\" DevicePath \"\""
Sep 09 21:10:24.182544 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:10:24.182330    2579 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e722d823-93a6-4495-9e58-0407ab0d8fd5-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-224.ec2.internal\" DevicePath \"\""
Sep 09 21:10:24.182544 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:10:24.182344    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2r7pm\" (UniqueName: \"kubernetes.io/projected/e722d823-93a6-4495-9e58-0407ab0d8fd5-kube-api-access-2r7pm\") on node \"ip-10-0-138-224.ec2.internal\" DevicePath \"\""
Sep 09 21:10:24.182544 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:10:24.182492    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d0c5c989-6102-415f-9bfc-faeb8eec7311-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-762cd-predictor-6b8d96d748-lw8dj\" (UID: \"d0c5c989-6102-415f-9bfc-faeb8eec7311\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-762cd-predictor-6b8d96d748-lw8dj"
Sep 09 21:10:24.182981 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:10:24.182954    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d0c5c989-6102-415f-9bfc-faeb8eec7311-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-762cd-predictor-6b8d96d748-lw8dj\" (UID: \"d0c5c989-6102-415f-9bfc-faeb8eec7311\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-762cd-predictor-6b8d96d748-lw8dj"
Sep 09 21:10:24.183097 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:10:24.182954    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d0c5c989-6102-415f-9bfc-faeb8eec7311-cabundle-cert\") pod \"isvc-sklearn-scale-raw-762cd-predictor-6b8d96d748-lw8dj\" (UID: \"d0c5c989-6102-415f-9bfc-faeb8eec7311\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-762cd-predictor-6b8d96d748-lw8dj"
Sep 09 21:10:24.184888 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:10:24.184871    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d0c5c989-6102-415f-9bfc-faeb8eec7311-proxy-tls\") pod \"isvc-sklearn-scale-raw-762cd-predictor-6b8d96d748-lw8dj\" (UID: \"d0c5c989-6102-415f-9bfc-faeb8eec7311\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-762cd-predictor-6b8d96d748-lw8dj"
Sep 09 21:10:24.192633 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:10:24.192615    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-79w4j\" (UniqueName: \"kubernetes.io/projected/d0c5c989-6102-415f-9bfc-faeb8eec7311-kube-api-access-79w4j\") pod \"isvc-sklearn-scale-raw-762cd-predictor-6b8d96d748-lw8dj\" (UID: \"d0c5c989-6102-415f-9bfc-faeb8eec7311\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-762cd-predictor-6b8d96d748-lw8dj"
Sep 09 21:10:24.216884 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:10:24.216858    2579 generic.go:356] "Generic (PLEG): container finished" podID="e722d823-93a6-4495-9e58-0407ab0d8fd5" containerID="cab9bc58008dcaedde5054741521a50b2d36a09e5f5ec653e16318451c05d771" exitCode=2
Sep 09 21:10:24.216884 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:10:24.216879    2579 generic.go:356] "Generic (PLEG): container finished" podID="e722d823-93a6-4495-9e58-0407ab0d8fd5" containerID="91f6c681920fd47dc1ad558b8355ec371b6cf5c0bcb7d7dfad4188eb8f7de5ef" exitCode=2
Sep 09 21:10:24.217057 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:10:24.216887    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-afe12-predictor-57759dbb49-s4j9t" event={"ID":"e722d823-93a6-4495-9e58-0407ab0d8fd5","Type":"ContainerDied","Data":"cab9bc58008dcaedde5054741521a50b2d36a09e5f5ec653e16318451c05d771"}
Sep 09 21:10:24.217057 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:10:24.216919    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-afe12-predictor-57759dbb49-s4j9t"
Sep 09 21:10:24.217057 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:10:24.216990    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-afe12-predictor-57759dbb49-s4j9t" event={"ID":"e722d823-93a6-4495-9e58-0407ab0d8fd5","Type":"ContainerDied","Data":"91f6c681920fd47dc1ad558b8355ec371b6cf5c0bcb7d7dfad4188eb8f7de5ef"}
Sep 09 21:10:24.217057 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:10:24.217005    2579 scope.go:117] "RemoveContainer" containerID="cab9bc58008dcaedde5054741521a50b2d36a09e5f5ec653e16318451c05d771"
Sep 09 21:10:24.217057 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:10:24.217028    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-afe12-predictor-57759dbb49-s4j9t" event={"ID":"e722d823-93a6-4495-9e58-0407ab0d8fd5","Type":"ContainerDied","Data":"98c404454f0b6d1a90b1ab2a077403b6197e53a98d3e1ff9976e2919c03a2c4f"}
Sep 09 21:10:24.220168 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:10:24.220054    2579 generic.go:356] "Generic (PLEG): container finished" podID="493338d2-24b1-47aa-a365-91e4d5274121" containerID="e06b2fc5d20647a961e9c8ca5c812ab4f278736a6d096bf085cd8463b12b51d8" exitCode=2
Sep 09 21:10:24.220168 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:10:24.220112    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-afe12-predictor-849d5f794b-6nl27" event={"ID":"493338d2-24b1-47aa-a365-91e4d5274121","Type":"ContainerDied","Data":"e06b2fc5d20647a961e9c8ca5c812ab4f278736a6d096bf085cd8463b12b51d8"}
Sep 09 21:10:24.225552 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:10:24.225523    2579 scope.go:117] "RemoveContainer" containerID="91f6c681920fd47dc1ad558b8355ec371b6cf5c0bcb7d7dfad4188eb8f7de5ef"
Sep 09 21:10:24.232622 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:10:24.232605    2579 scope.go:117] "RemoveContainer" containerID="cab9bc58008dcaedde5054741521a50b2d36a09e5f5ec653e16318451c05d771"
Sep 09 21:10:24.232913 ip-10-0-138-224 kubenswrapper[2579]: E0909 21:10:24.232896    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cab9bc58008dcaedde5054741521a50b2d36a09e5f5ec653e16318451c05d771\": container with ID starting with cab9bc58008dcaedde5054741521a50b2d36a09e5f5ec653e16318451c05d771 not found: ID does not exist" containerID="cab9bc58008dcaedde5054741521a50b2d36a09e5f5ec653e16318451c05d771"
Sep 09 21:10:24.232982 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:10:24.232925    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cab9bc58008dcaedde5054741521a50b2d36a09e5f5ec653e16318451c05d771"} err="failed to get container status \"cab9bc58008dcaedde5054741521a50b2d36a09e5f5ec653e16318451c05d771\": rpc error: code = NotFound desc = could not find container \"cab9bc58008dcaedde5054741521a50b2d36a09e5f5ec653e16318451c05d771\": container with ID starting with cab9bc58008dcaedde5054741521a50b2d36a09e5f5ec653e16318451c05d771 not found: ID does not exist"
Sep 09 21:10:24.232982 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:10:24.232947    2579 scope.go:117] "RemoveContainer" containerID="91f6c681920fd47dc1ad558b8355ec371b6cf5c0bcb7d7dfad4188eb8f7de5ef"
Sep 09 21:10:24.233211 ip-10-0-138-224 kubenswrapper[2579]: E0909 21:10:24.233194    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91f6c681920fd47dc1ad558b8355ec371b6cf5c0bcb7d7dfad4188eb8f7de5ef\": container with ID starting with 91f6c681920fd47dc1ad558b8355ec371b6cf5c0bcb7d7dfad4188eb8f7de5ef not found: ID does not exist" containerID="91f6c681920fd47dc1ad558b8355ec371b6cf5c0bcb7d7dfad4188eb8f7de5ef"
Sep 09 21:10:24.233251 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:10:24.233218    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91f6c681920fd47dc1ad558b8355ec371b6cf5c0bcb7d7dfad4188eb8f7de5ef"} err="failed to get container status \"91f6c681920fd47dc1ad558b8355ec371b6cf5c0bcb7d7dfad4188eb8f7de5ef\": rpc error: code = NotFound desc = could not find container \"91f6c681920fd47dc1ad558b8355ec371b6cf5c0bcb7d7dfad4188eb8f7de5ef\": container with ID starting with 91f6c681920fd47dc1ad558b8355ec371b6cf5c0bcb7d7dfad4188eb8f7de5ef not found: ID does not exist"
Sep 09 21:10:24.233251 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:10:24.233233    2579 scope.go:117] "RemoveContainer" containerID="cab9bc58008dcaedde5054741521a50b2d36a09e5f5ec653e16318451c05d771"
Sep 09 21:10:24.233484 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:10:24.233429    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cab9bc58008dcaedde5054741521a50b2d36a09e5f5ec653e16318451c05d771"} err="failed to get container status \"cab9bc58008dcaedde5054741521a50b2d36a09e5f5ec653e16318451c05d771\": rpc error: code = NotFound desc = could not find container \"cab9bc58008dcaedde5054741521a50b2d36a09e5f5ec653e16318451c05d771\": container with ID starting with cab9bc58008dcaedde5054741521a50b2d36a09e5f5ec653e16318451c05d771 not found: ID does not exist"
Sep 09 21:10:24.233540 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:10:24.233489    2579 scope.go:117] "RemoveContainer" containerID="91f6c681920fd47dc1ad558b8355ec371b6cf5c0bcb7d7dfad4188eb8f7de5ef"
Sep 09 21:10:24.233723 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:10:24.233706    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91f6c681920fd47dc1ad558b8355ec371b6cf5c0bcb7d7dfad4188eb8f7de5ef"} err="failed to get container status \"91f6c681920fd47dc1ad558b8355ec371b6cf5c0bcb7d7dfad4188eb8f7de5ef\": rpc error: code = NotFound desc = could not find container \"91f6c681920fd47dc1ad558b8355ec371b6cf5c0bcb7d7dfad4188eb8f7de5ef\": container with ID starting with 91f6c681920fd47dc1ad558b8355ec371b6cf5c0bcb7d7dfad4188eb8f7de5ef not found: ID does not exist"
Sep 09 21:10:24.239755 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:10:24.239735    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-afe12-predictor-57759dbb49-s4j9t"]
Sep 09 21:10:24.243286 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:10:24.243267    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-afe12-predictor-57759dbb49-s4j9t"]
Sep 09 21:10:24.288749 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:10:24.288724    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-762cd-predictor-6b8d96d748-lw8dj"
Sep 09 21:10:24.567631 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:10:24.567605    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-762cd-predictor-6b8d96d748-lw8dj"]
Sep 09 21:10:24.569305 ip-10-0-138-224 kubenswrapper[2579]: W0909 21:10:24.569282    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0c5c989_6102_415f_9bfc_faeb8eec7311.slice/crio-2e817adbd668174a1ddea8d71e04e4c006aa65fe9fa385596e6591a71ba03d58 WatchSource:0}: Error finding container 2e817adbd668174a1ddea8d71e04e4c006aa65fe9fa385596e6591a71ba03d58: Status 404 returned error can't find the container with id 2e817adbd668174a1ddea8d71e04e4c006aa65fe9fa385596e6591a71ba03d58
Sep 09 21:10:25.227702 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:10:25.227664    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-762cd-predictor-6b8d96d748-lw8dj" event={"ID":"d0c5c989-6102-415f-9bfc-faeb8eec7311","Type":"ContainerStarted","Data":"5cb094094252f4ee991eb2e0862e2f1ba7019dcd049cab8ca0a665c8d1f65583"}
Sep 09 21:10:25.227885 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:10:25.227744    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-762cd-predictor-6b8d96d748-lw8dj" event={"ID":"d0c5c989-6102-415f-9bfc-faeb8eec7311","Type":"ContainerStarted","Data":"2e817adbd668174a1ddea8d71e04e4c006aa65fe9fa385596e6591a71ba03d58"}
Sep 09 21:10:25.291497 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:10:25.291458    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e722d823-93a6-4495-9e58-0407ab0d8fd5" path="/var/lib/kubelet/pods/e722d823-93a6-4495-9e58-0407ab0d8fd5/volumes"
Sep 09 21:10:25.798901 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:10:25.798863    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-afe12-predictor-849d5f794b-6nl27" podUID="493338d2-24b1-47aa-a365-91e4d5274121" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.39:8643/healthz\": dial tcp 10.134.0.39:8643: connect: connection refused"
Sep 09 21:10:29.249913 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:10:29.249882    2579 generic.go:356] "Generic (PLEG): container finished" podID="493338d2-24b1-47aa-a365-91e4d5274121" containerID="c98f01b246a7f53db638e0dabaa8a181031ee40d716ec8a8c9f61cebafe370bf" exitCode=0
Sep 09 21:10:29.250323 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:10:29.249951    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-afe12-predictor-849d5f794b-6nl27" event={"ID":"493338d2-24b1-47aa-a365-91e4d5274121","Type":"ContainerDied","Data":"c98f01b246a7f53db638e0dabaa8a181031ee40d716ec8a8c9f61cebafe370bf"}
Sep 09 21:10:29.251311 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:10:29.251291    2579 generic.go:356] "Generic (PLEG): container finished" podID="d0c5c989-6102-415f-9bfc-faeb8eec7311" containerID="5cb094094252f4ee991eb2e0862e2f1ba7019dcd049cab8ca0a665c8d1f65583" exitCode=0
Sep 09 21:10:29.251368 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:10:29.251319    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-762cd-predictor-6b8d96d748-lw8dj" event={"ID":"d0c5c989-6102-415f-9bfc-faeb8eec7311","Type":"ContainerDied","Data":"5cb094094252f4ee991eb2e0862e2f1ba7019dcd049cab8ca0a665c8d1f65583"}
Sep 09 21:10:30.258834 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:10:30.258799    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-762cd-predictor-6b8d96d748-lw8dj" event={"ID":"d0c5c989-6102-415f-9bfc-faeb8eec7311","Type":"ContainerStarted","Data":"adc2d4842367b89720894d97807de9a06fb82041c3a9b1dcecce01e3a4f87271"}
Sep 09 21:10:30.259239 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:10:30.258863    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-762cd-predictor-6b8d96d748-lw8dj" event={"ID":"d0c5c989-6102-415f-9bfc-faeb8eec7311","Type":"ContainerStarted","Data":"67ce1ba30caea136f1add0232d38643138a6e3073856a83ce3fe087fda05da85"}
Sep 09 21:10:30.259239 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:10:30.259076    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-762cd-predictor-6b8d96d748-lw8dj"
Sep 09 21:10:30.278703 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:10:30.278655    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-762cd-predictor-6b8d96d748-lw8dj" podStartSLOduration=3.596551584 podStartE2EDuration="7.278639255s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.682087671s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:10:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:10:30.277871886 +0000 UTC m=+1087.517966007" watchObservedRunningTime="2026-09-09 21:10:30.278639255 +0000 UTC m=+1087.518733373"
Sep 09 21:10:30.799007 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:10:30.798972    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-afe12-predictor-849d5f794b-6nl27" podUID="493338d2-24b1-47aa-a365-91e4d5274121" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.39:8643/healthz\": dial tcp 10.134.0.39:8643: connect: connection refused"
Sep 09 21:10:30.804497 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:10:30.804434    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-afe12-predictor-849d5f794b-6nl27" podUID="493338d2-24b1-47aa-a365-91e4d5274121" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Sep 09 21:10:30.804799 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:10:30.804776    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-afe12-predictor-849d5f794b-6nl27" podUID="493338d2-24b1-47aa-a365-91e4d5274121" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 21:10:31.263981 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:10:31.263951    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-762cd-predictor-6b8d96d748-lw8dj"
Sep 09 21:10:31.265366 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:10:31.265336    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-762cd-predictor-6b8d96d748-lw8dj" podUID="d0c5c989-6102-415f-9bfc-faeb8eec7311" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Sep 09 21:10:32.269416 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:10:32.269381    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-762cd-predictor-6b8d96d748-lw8dj" podUID="d0c5c989-6102-415f-9bfc-faeb8eec7311" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Sep 09 21:10:35.799313 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:10:35.799260    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-afe12-predictor-849d5f794b-6nl27" podUID="493338d2-24b1-47aa-a365-91e4d5274121" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.39:8643/healthz\": dial tcp 10.134.0.39:8643: connect: connection refused"
Sep 09 21:10:35.799797 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:10:35.799421    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-afe12-predictor-849d5f794b-6nl27"
Sep 09 21:10:37.273850 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:10:37.273820    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-762cd-predictor-6b8d96d748-lw8dj"
Sep 09 21:10:37.274343 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:10:37.274318    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-762cd-predictor-6b8d96d748-lw8dj" podUID="d0c5c989-6102-415f-9bfc-faeb8eec7311" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Sep 09 21:10:40.798665 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:10:40.798624    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-afe12-predictor-849d5f794b-6nl27" podUID="493338d2-24b1-47aa-a365-91e4d5274121" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.39:8643/healthz\": dial tcp 10.134.0.39:8643: connect: connection refused"
Sep 09 21:10:40.804022 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:10:40.803992    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-afe12-predictor-849d5f794b-6nl27" podUID="493338d2-24b1-47aa-a365-91e4d5274121" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Sep 09 21:10:40.804339 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:10:40.804317    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-afe12-predictor-849d5f794b-6nl27" podUID="493338d2-24b1-47aa-a365-91e4d5274121" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 21:10:45.799782 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:10:45.799731    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-afe12-predictor-849d5f794b-6nl27" podUID="493338d2-24b1-47aa-a365-91e4d5274121" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.39:8643/healthz\": dial tcp 10.134.0.39:8643: connect: connection refused"
Sep 09 21:10:47.275004 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:10:47.274962    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-762cd-predictor-6b8d96d748-lw8dj" podUID="d0c5c989-6102-415f-9bfc-faeb8eec7311" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Sep 09 21:10:50.799782 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:10:50.799732    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-afe12-predictor-849d5f794b-6nl27" podUID="493338d2-24b1-47aa-a365-91e4d5274121" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.39:8643/healthz\": dial tcp 10.134.0.39:8643: connect: connection refused"
Sep 09 21:10:50.804150 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:10:50.804117    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-afe12-predictor-849d5f794b-6nl27" podUID="493338d2-24b1-47aa-a365-91e4d5274121" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Sep 09 21:10:50.804304 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:10:50.804255    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-afe12-predictor-849d5f794b-6nl27"
Sep 09 21:10:50.804510 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:10:50.804485    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-afe12-predictor-849d5f794b-6nl27" podUID="493338d2-24b1-47aa-a365-91e4d5274121" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 21:10:50.804605 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:10:50.804593    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-afe12-predictor-849d5f794b-6nl27"
Sep 09 21:10:53.847588 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:10:53.847566    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-afe12-predictor-849d5f794b-6nl27"
Sep 09 21:10:53.928407 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:10:53.928367    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/493338d2-24b1-47aa-a365-91e4d5274121-cabundle-cert\") pod \"493338d2-24b1-47aa-a365-91e4d5274121\" (UID: \"493338d2-24b1-47aa-a365-91e4d5274121\") "
Sep 09 21:10:53.928597 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:10:53.928424    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/493338d2-24b1-47aa-a365-91e4d5274121-kserve-provision-location\") pod \"493338d2-24b1-47aa-a365-91e4d5274121\" (UID: \"493338d2-24b1-47aa-a365-91e4d5274121\") "
Sep 09 21:10:53.928597 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:10:53.928483    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/493338d2-24b1-47aa-a365-91e4d5274121-proxy-tls\") pod \"493338d2-24b1-47aa-a365-91e4d5274121\" (UID: \"493338d2-24b1-47aa-a365-91e4d5274121\") "
Sep 09 21:10:53.928597 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:10:53.928511    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/493338d2-24b1-47aa-a365-91e4d5274121-kube-rbac-proxy-sar-config\") pod \"493338d2-24b1-47aa-a365-91e4d5274121\" (UID: \"493338d2-24b1-47aa-a365-91e4d5274121\") "
Sep 09 21:10:53.928597 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:10:53.928538    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mkrp5\" (UniqueName: \"kubernetes.io/projected/493338d2-24b1-47aa-a365-91e4d5274121-kube-api-access-mkrp5\") pod \"493338d2-24b1-47aa-a365-91e4d5274121\" (UID: \"493338d2-24b1-47aa-a365-91e4d5274121\") "
Sep 09 21:10:53.928827 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:10:53.928785    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/493338d2-24b1-47aa-a365-91e4d5274121-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "493338d2-24b1-47aa-a365-91e4d5274121" (UID: "493338d2-24b1-47aa-a365-91e4d5274121"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:10:53.928880 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:10:53.928822    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/493338d2-24b1-47aa-a365-91e4d5274121-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "493338d2-24b1-47aa-a365-91e4d5274121" (UID: "493338d2-24b1-47aa-a365-91e4d5274121"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:10:53.928925 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:10:53.928888    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/493338d2-24b1-47aa-a365-91e4d5274121-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "493338d2-24b1-47aa-a365-91e4d5274121" (UID: "493338d2-24b1-47aa-a365-91e4d5274121"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:10:53.930743 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:10:53.930721    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/493338d2-24b1-47aa-a365-91e4d5274121-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "493338d2-24b1-47aa-a365-91e4d5274121" (UID: "493338d2-24b1-47aa-a365-91e4d5274121"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:10:53.930860 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:10:53.930814    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/493338d2-24b1-47aa-a365-91e4d5274121-kube-api-access-mkrp5" (OuterVolumeSpecName: "kube-api-access-mkrp5") pod "493338d2-24b1-47aa-a365-91e4d5274121" (UID: "493338d2-24b1-47aa-a365-91e4d5274121"). InnerVolumeSpecName "kube-api-access-mkrp5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:10:54.029422 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:10:54.029343    2579 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/493338d2-24b1-47aa-a365-91e4d5274121-cabundle-cert\") on node \"ip-10-0-138-224.ec2.internal\" DevicePath \"\""
Sep 09 21:10:54.029422 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:10:54.029370    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/493338d2-24b1-47aa-a365-91e4d5274121-kserve-provision-location\") on node \"ip-10-0-138-224.ec2.internal\" DevicePath \"\""
Sep 09 21:10:54.029422 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:10:54.029380    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/493338d2-24b1-47aa-a365-91e4d5274121-proxy-tls\") on node \"ip-10-0-138-224.ec2.internal\" DevicePath \"\""
Sep 09 21:10:54.029422 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:10:54.029388    2579 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/493338d2-24b1-47aa-a365-91e4d5274121-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-224.ec2.internal\" DevicePath \"\""
Sep 09 21:10:54.029422 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:10:54.029396    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mkrp5\" (UniqueName: \"kubernetes.io/projected/493338d2-24b1-47aa-a365-91e4d5274121-kube-api-access-mkrp5\") on node \"ip-10-0-138-224.ec2.internal\" DevicePath \"\""
Sep 09 21:10:54.371431 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:10:54.371348    2579 generic.go:356] "Generic (PLEG): container finished" podID="493338d2-24b1-47aa-a365-91e4d5274121" containerID="969e93e538e009544ad055167ae8f00f43d92c446069f2ab83909f2478f8c7a1" exitCode=0
Sep 09 21:10:54.371431 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:10:54.371411    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-afe12-predictor-849d5f794b-6nl27"
Sep 09 21:10:54.371642 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:10:54.371432    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-afe12-predictor-849d5f794b-6nl27" event={"ID":"493338d2-24b1-47aa-a365-91e4d5274121","Type":"ContainerDied","Data":"969e93e538e009544ad055167ae8f00f43d92c446069f2ab83909f2478f8c7a1"}
Sep 09 21:10:54.371642 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:10:54.371510    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-afe12-predictor-849d5f794b-6nl27" event={"ID":"493338d2-24b1-47aa-a365-91e4d5274121","Type":"ContainerDied","Data":"64a73eee7c676947be527f83aa41aa3048d58869caacc38c603bca8e08dab80c"}
Sep 09 21:10:54.371642 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:10:54.371543    2579 scope.go:117] "RemoveContainer" containerID="969e93e538e009544ad055167ae8f00f43d92c446069f2ab83909f2478f8c7a1"
Sep 09 21:10:54.380546 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:10:54.380484    2579 scope.go:117] "RemoveContainer" containerID="e06b2fc5d20647a961e9c8ca5c812ab4f278736a6d096bf085cd8463b12b51d8"
Sep 09 21:10:54.388877 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:10:54.388860    2579 scope.go:117] "RemoveContainer" containerID="c98f01b246a7f53db638e0dabaa8a181031ee40d716ec8a8c9f61cebafe370bf"
Sep 09 21:10:54.395920 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:10:54.395899    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-afe12-predictor-849d5f794b-6nl27"]
Sep 09 21:10:54.396699 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:10:54.396679    2579 scope.go:117] "RemoveContainer" containerID="3070aa591a4785816c528f158a71f01be4e90628f65f9a37e383d9ad55e76e19"
Sep 09 21:10:54.400870 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:10:54.400850    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-afe12-predictor-849d5f794b-6nl27"]
Sep 09 21:10:54.404712 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:10:54.404696    2579 scope.go:117] "RemoveContainer" containerID="969e93e538e009544ad055167ae8f00f43d92c446069f2ab83909f2478f8c7a1"
Sep 09 21:10:54.404983 ip-10-0-138-224 kubenswrapper[2579]: E0909 21:10:54.404964    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"969e93e538e009544ad055167ae8f00f43d92c446069f2ab83909f2478f8c7a1\": container with ID starting with 969e93e538e009544ad055167ae8f00f43d92c446069f2ab83909f2478f8c7a1 not found: ID does not exist" containerID="969e93e538e009544ad055167ae8f00f43d92c446069f2ab83909f2478f8c7a1"
Sep 09 21:10:54.405047 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:10:54.404993    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"969e93e538e009544ad055167ae8f00f43d92c446069f2ab83909f2478f8c7a1"} err="failed to get container status \"969e93e538e009544ad055167ae8f00f43d92c446069f2ab83909f2478f8c7a1\": rpc error: code = NotFound desc = could not find container \"969e93e538e009544ad055167ae8f00f43d92c446069f2ab83909f2478f8c7a1\": container with ID starting with 969e93e538e009544ad055167ae8f00f43d92c446069f2ab83909f2478f8c7a1 not found: ID does not exist"
Sep 09 21:10:54.405047 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:10:54.405010    2579 scope.go:117] "RemoveContainer" containerID="e06b2fc5d20647a961e9c8ca5c812ab4f278736a6d096bf085cd8463b12b51d8"
Sep 09 21:10:54.405233 ip-10-0-138-224 kubenswrapper[2579]: E0909 21:10:54.405217    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e06b2fc5d20647a961e9c8ca5c812ab4f278736a6d096bf085cd8463b12b51d8\": container with ID starting with e06b2fc5d20647a961e9c8ca5c812ab4f278736a6d096bf085cd8463b12b51d8 not found: ID does not exist" containerID="e06b2fc5d20647a961e9c8ca5c812ab4f278736a6d096bf085cd8463b12b51d8"
Sep 09 21:10:54.405276 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:10:54.405236    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e06b2fc5d20647a961e9c8ca5c812ab4f278736a6d096bf085cd8463b12b51d8"} err="failed to get container status \"e06b2fc5d20647a961e9c8ca5c812ab4f278736a6d096bf085cd8463b12b51d8\": rpc error: code = NotFound desc = could not find container \"e06b2fc5d20647a961e9c8ca5c812ab4f278736a6d096bf085cd8463b12b51d8\": container with ID starting with e06b2fc5d20647a961e9c8ca5c812ab4f278736a6d096bf085cd8463b12b51d8 not found: ID does not exist"
Sep 09 21:10:54.405276 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:10:54.405248    2579 scope.go:117] "RemoveContainer" containerID="c98f01b246a7f53db638e0dabaa8a181031ee40d716ec8a8c9f61cebafe370bf"
Sep 09 21:10:54.405547 ip-10-0-138-224 kubenswrapper[2579]: E0909 21:10:54.405531    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c98f01b246a7f53db638e0dabaa8a181031ee40d716ec8a8c9f61cebafe370bf\": container with ID starting with c98f01b246a7f53db638e0dabaa8a181031ee40d716ec8a8c9f61cebafe370bf not found: ID does not exist" containerID="c98f01b246a7f53db638e0dabaa8a181031ee40d716ec8a8c9f61cebafe370bf"
Sep 09 21:10:54.405606 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:10:54.405550    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c98f01b246a7f53db638e0dabaa8a181031ee40d716ec8a8c9f61cebafe370bf"} err="failed to get container status \"c98f01b246a7f53db638e0dabaa8a181031ee40d716ec8a8c9f61cebafe370bf\": rpc error: code = NotFound desc = could not find container \"c98f01b246a7f53db638e0dabaa8a181031ee40d716ec8a8c9f61cebafe370bf\": container with ID starting with c98f01b246a7f53db638e0dabaa8a181031ee40d716ec8a8c9f61cebafe370bf not found: ID does not exist"
Sep 09 21:10:54.405606 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:10:54.405562    2579 scope.go:117] "RemoveContainer" containerID="3070aa591a4785816c528f158a71f01be4e90628f65f9a37e383d9ad55e76e19"
Sep 09 21:10:54.405769 ip-10-0-138-224 kubenswrapper[2579]: E0909 21:10:54.405752    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3070aa591a4785816c528f158a71f01be4e90628f65f9a37e383d9ad55e76e19\": container with ID starting with 3070aa591a4785816c528f158a71f01be4e90628f65f9a37e383d9ad55e76e19 not found: ID does not exist" containerID="3070aa591a4785816c528f158a71f01be4e90628f65f9a37e383d9ad55e76e19"
Sep 09 21:10:54.405822 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:10:54.405784    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3070aa591a4785816c528f158a71f01be4e90628f65f9a37e383d9ad55e76e19"} err="failed to get container status \"3070aa591a4785816c528f158a71f01be4e90628f65f9a37e383d9ad55e76e19\": rpc error: code = NotFound desc = could not find container \"3070aa591a4785816c528f158a71f01be4e90628f65f9a37e383d9ad55e76e19\": container with ID starting with 3070aa591a4785816c528f158a71f01be4e90628f65f9a37e383d9ad55e76e19 not found: ID does not exist"
Sep 09 21:10:55.290776 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:10:55.290742    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="493338d2-24b1-47aa-a365-91e4d5274121" path="/var/lib/kubelet/pods/493338d2-24b1-47aa-a365-91e4d5274121/volumes"
Sep 09 21:10:57.275065 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:10:57.275023    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-762cd-predictor-6b8d96d748-lw8dj" podUID="d0c5c989-6102-415f-9bfc-faeb8eec7311" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Sep 09 21:11:07.274802 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:11:07.274763    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-762cd-predictor-6b8d96d748-lw8dj" podUID="d0c5c989-6102-415f-9bfc-faeb8eec7311" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Sep 09 21:11:17.274285 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:11:17.274242    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-762cd-predictor-6b8d96d748-lw8dj" podUID="d0c5c989-6102-415f-9bfc-faeb8eec7311" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Sep 09 21:11:27.275224 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:11:27.275184    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-762cd-predictor-6b8d96d748-lw8dj" podUID="d0c5c989-6102-415f-9bfc-faeb8eec7311" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Sep 09 21:11:37.274805 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:11:37.274764    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-762cd-predictor-6b8d96d748-lw8dj" podUID="d0c5c989-6102-415f-9bfc-faeb8eec7311" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Sep 09 21:11:47.275277 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:11:47.275195    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-762cd-predictor-6b8d96d748-lw8dj" podUID="d0c5c989-6102-415f-9bfc-faeb8eec7311" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Sep 09 21:11:48.285563 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:11:48.285521    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-762cd-predictor-6b8d96d748-lw8dj" podUID="d0c5c989-6102-415f-9bfc-faeb8eec7311" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Sep 09 21:11:58.285613 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:11:58.285567    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-762cd-predictor-6b8d96d748-lw8dj" podUID="d0c5c989-6102-415f-9bfc-faeb8eec7311" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Sep 09 21:12:08.285839 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:12:08.285789    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-762cd-predictor-6b8d96d748-lw8dj" podUID="d0c5c989-6102-415f-9bfc-faeb8eec7311" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Sep 09 21:12:18.285612 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:12:18.285565    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-762cd-predictor-6b8d96d748-lw8dj" podUID="d0c5c989-6102-415f-9bfc-faeb8eec7311" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Sep 09 21:12:28.285601 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:12:28.285542    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-762cd-predictor-6b8d96d748-lw8dj" podUID="d0c5c989-6102-415f-9bfc-faeb8eec7311" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Sep 09 21:12:38.285990 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:12:38.285949    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-762cd-predictor-6b8d96d748-lw8dj" podUID="d0c5c989-6102-415f-9bfc-faeb8eec7311" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Sep 09 21:12:48.285933 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:12:48.285883    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-762cd-predictor-6b8d96d748-lw8dj" podUID="d0c5c989-6102-415f-9bfc-faeb8eec7311" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Sep 09 21:12:58.285599 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:12:58.285556    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-762cd-predictor-6b8d96d748-lw8dj" podUID="d0c5c989-6102-415f-9bfc-faeb8eec7311" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Sep 09 21:13:08.286038 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:13:08.286002    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-762cd-predictor-6b8d96d748-lw8dj"
Sep 09 21:13:13.912914 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:13:13.912882    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-762cd-predictor-6b8d96d748-lw8dj"]
Sep 09 21:13:13.913294 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:13:13.913145    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-762cd-predictor-6b8d96d748-lw8dj" podUID="d0c5c989-6102-415f-9bfc-faeb8eec7311" containerName="kserve-container" containerID="cri-o://67ce1ba30caea136f1add0232d38643138a6e3073856a83ce3fe087fda05da85" gracePeriod=30
Sep 09 21:13:13.913294 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:13:13.913217    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-762cd-predictor-6b8d96d748-lw8dj" podUID="d0c5c989-6102-415f-9bfc-faeb8eec7311" containerName="kube-rbac-proxy" containerID="cri-o://adc2d4842367b89720894d97807de9a06fb82041c3a9b1dcecce01e3a4f87271" gracePeriod=30
Sep 09 21:13:14.995195 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:13:14.995165    2579 generic.go:356] "Generic (PLEG): container finished" podID="d0c5c989-6102-415f-9bfc-faeb8eec7311" containerID="adc2d4842367b89720894d97807de9a06fb82041c3a9b1dcecce01e3a4f87271" exitCode=2
Sep 09 21:13:14.995568 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:13:14.995238    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-762cd-predictor-6b8d96d748-lw8dj" event={"ID":"d0c5c989-6102-415f-9bfc-faeb8eec7311","Type":"ContainerDied","Data":"adc2d4842367b89720894d97807de9a06fb82041c3a9b1dcecce01e3a4f87271"}
Sep 09 21:13:17.270333 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:13:17.270292    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-762cd-predictor-6b8d96d748-lw8dj" podUID="d0c5c989-6102-415f-9bfc-faeb8eec7311" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.41:8643/healthz\": dial tcp 10.134.0.41:8643: connect: connection refused"
Sep 09 21:13:18.286519 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:13:18.286379    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-762cd-predictor-6b8d96d748-lw8dj" podUID="d0c5c989-6102-415f-9bfc-faeb8eec7311" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Sep 09 21:13:19.107417 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:13:19.107385    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5cd44d6886-s2gkb"]
Sep 09 21:13:19.107754 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:13:19.107741    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="493338d2-24b1-47aa-a365-91e4d5274121" containerName="kube-rbac-proxy"
Sep 09 21:13:19.107807 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:13:19.107756    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="493338d2-24b1-47aa-a365-91e4d5274121" containerName="kube-rbac-proxy"
Sep 09 21:13:19.107807 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:13:19.107772    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e722d823-93a6-4495-9e58-0407ab0d8fd5" containerName="kserve-container"
Sep 09 21:13:19.107807 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:13:19.107777    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="e722d823-93a6-4495-9e58-0407ab0d8fd5" containerName="kserve-container"
Sep 09 21:13:19.107807 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:13:19.107786    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="493338d2-24b1-47aa-a365-91e4d5274121" containerName="agent"
Sep 09 21:13:19.107807 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:13:19.107793    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="493338d2-24b1-47aa-a365-91e4d5274121" containerName="agent"
Sep 09 21:13:19.107807 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:13:19.107802    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="493338d2-24b1-47aa-a365-91e4d5274121" containerName="storage-initializer"
Sep 09 21:13:19.107807 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:13:19.107807    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="493338d2-24b1-47aa-a365-91e4d5274121" containerName="storage-initializer"
Sep 09 21:13:19.108051 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:13:19.107824    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e722d823-93a6-4495-9e58-0407ab0d8fd5" containerName="kube-rbac-proxy"
Sep 09 21:13:19.108051 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:13:19.107829    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="e722d823-93a6-4495-9e58-0407ab0d8fd5" containerName="kube-rbac-proxy"
Sep 09 21:13:19.108051 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:13:19.107834    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="493338d2-24b1-47aa-a365-91e4d5274121" containerName="kserve-container"
Sep 09 21:13:19.108051 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:13:19.107839    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="493338d2-24b1-47aa-a365-91e4d5274121" containerName="kserve-container"
Sep 09 21:13:19.108051 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:13:19.107903    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="493338d2-24b1-47aa-a365-91e4d5274121" containerName="kserve-container"
Sep 09 21:13:19.108051 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:13:19.107910    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="e722d823-93a6-4495-9e58-0407ab0d8fd5" containerName="kserve-container"
Sep 09 21:13:19.108051 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:13:19.107917    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="e722d823-93a6-4495-9e58-0407ab0d8fd5" containerName="kube-rbac-proxy"
Sep 09 21:13:19.108051 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:13:19.107923    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="493338d2-24b1-47aa-a365-91e4d5274121" containerName="kube-rbac-proxy"
Sep 09 21:13:19.108051 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:13:19.107929    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="493338d2-24b1-47aa-a365-91e4d5274121" containerName="agent"
Sep 09 21:13:19.110917 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:13:19.110891    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5cd44d6886-s2gkb"
Sep 09 21:13:19.114717 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:13:19.114694    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-v2-predictor-serving-cert\""
Sep 09 21:13:19.114841 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:13:19.114825    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-kube-rbac-proxy-sar-config\""
Sep 09 21:13:19.127456 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:13:19.127415    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5cd44d6886-s2gkb"]
Sep 09 21:13:19.155765 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:13:19.155733    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-pv7tx"]
Sep 09 21:13:19.159423 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:13:19.159402    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-pv7tx"
Sep 09 21:13:19.162239 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:13:19.162218    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-predictor-serving-cert\""
Sep 09 21:13:19.170472 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:13:19.170429    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-pv7tx"]
Sep 09 21:13:19.228700 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:13:19.228671    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dbcd4317-6eac-4497-aa10-5a2867fb1197-proxy-tls\") pod \"isvc-canary-create-v2-predictor-5cd44d6886-s2gkb\" (UID: \"dbcd4317-6eac-4497-aa10-5a2867fb1197\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5cd44d6886-s2gkb"
Sep 09 21:13:19.228868 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:13:19.228732    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/31dd0d17-13fe-4305-88eb-77c3636749cd-proxy-tls\") pod \"isvc-canary-create-predictor-854596548c-pv7tx\" (UID: \"31dd0d17-13fe-4305-88eb-77c3636749cd\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-pv7tx"
Sep 09 21:13:19.228868 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:13:19.228796    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dbcd4317-6eac-4497-aa10-5a2867fb1197-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-5cd44d6886-s2gkb\" (UID: \"dbcd4317-6eac-4497-aa10-5a2867fb1197\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5cd44d6886-s2gkb"
Sep 09 21:13:19.228868 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:13:19.228838    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7ssv\" (UniqueName: \"kubernetes.io/projected/31dd0d17-13fe-4305-88eb-77c3636749cd-kube-api-access-w7ssv\") pod \"isvc-canary-create-predictor-854596548c-pv7tx\" (UID: \"31dd0d17-13fe-4305-88eb-77c3636749cd\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-pv7tx"
Sep 09 21:13:19.228868 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:13:19.228868    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dbcd4317-6eac-4497-aa10-5a2867fb1197-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-5cd44d6886-s2gkb\" (UID: \"dbcd4317-6eac-4497-aa10-5a2867fb1197\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5cd44d6886-s2gkb"
Sep 09 21:13:19.229030 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:13:19.228956    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lnj9r\" (UniqueName: \"kubernetes.io/projected/dbcd4317-6eac-4497-aa10-5a2867fb1197-kube-api-access-lnj9r\") pod \"isvc-canary-create-v2-predictor-5cd44d6886-s2gkb\" (UID: \"dbcd4317-6eac-4497-aa10-5a2867fb1197\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5cd44d6886-s2gkb"
Sep 09 21:13:19.229030 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:13:19.228994    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dbcd4317-6eac-4497-aa10-5a2867fb1197-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-5cd44d6886-s2gkb\" (UID: \"dbcd4317-6eac-4497-aa10-5a2867fb1197\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5cd44d6886-s2gkb"
Sep 09 21:13:19.229098 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:13:19.229035    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/31dd0d17-13fe-4305-88eb-77c3636749cd-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-854596548c-pv7tx\" (UID: \"31dd0d17-13fe-4305-88eb-77c3636749cd\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-pv7tx"
Sep 09 21:13:19.229098 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:13:19.229053    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/31dd0d17-13fe-4305-88eb-77c3636749cd-kserve-provision-location\") pod \"isvc-canary-create-predictor-854596548c-pv7tx\" (UID: \"31dd0d17-13fe-4305-88eb-77c3636749cd\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-pv7tx"
Sep 09 21:13:19.229098 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:13:19.229072    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/31dd0d17-13fe-4305-88eb-77c3636749cd-cabundle-cert\") pod \"isvc-canary-create-predictor-854596548c-pv7tx\" (UID: \"31dd0d17-13fe-4305-88eb-77c3636749cd\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-pv7tx"
Sep 09 21:13:19.329791 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:13:19.329752    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/31dd0d17-13fe-4305-88eb-77c3636749cd-proxy-tls\") pod \"isvc-canary-create-predictor-854596548c-pv7tx\" (UID: \"31dd0d17-13fe-4305-88eb-77c3636749cd\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-pv7tx"
Sep 09 21:13:19.329791 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:13:19.329795    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dbcd4317-6eac-4497-aa10-5a2867fb1197-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-5cd44d6886-s2gkb\" (UID: \"dbcd4317-6eac-4497-aa10-5a2867fb1197\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5cd44d6886-s2gkb"
Sep 09 21:13:19.330321 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:13:19.329835    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7ssv\" (UniqueName: \"kubernetes.io/projected/31dd0d17-13fe-4305-88eb-77c3636749cd-kube-api-access-w7ssv\") pod \"isvc-canary-create-predictor-854596548c-pv7tx\" (UID: \"31dd0d17-13fe-4305-88eb-77c3636749cd\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-pv7tx"
Sep 09 21:13:19.330321 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:13:19.329871    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dbcd4317-6eac-4497-aa10-5a2867fb1197-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-5cd44d6886-s2gkb\" (UID: \"dbcd4317-6eac-4497-aa10-5a2867fb1197\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5cd44d6886-s2gkb"
Sep 09 21:13:19.330321 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:13:19.329905    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnj9r\" (UniqueName: \"kubernetes.io/projected/dbcd4317-6eac-4497-aa10-5a2867fb1197-kube-api-access-lnj9r\") pod \"isvc-canary-create-v2-predictor-5cd44d6886-s2gkb\" (UID: \"dbcd4317-6eac-4497-aa10-5a2867fb1197\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5cd44d6886-s2gkb"
Sep 09 21:13:19.330321 ip-10-0-138-224 kubenswrapper[2579]: E0909 21:13:19.329915    2579 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-create-predictor-serving-cert: secret "isvc-canary-create-predictor-serving-cert" not found
Sep 09 21:13:19.330321 ip-10-0-138-224 kubenswrapper[2579]: E0909 21:13:19.329981    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/31dd0d17-13fe-4305-88eb-77c3636749cd-proxy-tls podName:31dd0d17-13fe-4305-88eb-77c3636749cd nodeName:}" failed. No retries permitted until 2026-09-09 21:13:19.829961179 +0000 UTC m=+1257.070055297 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/31dd0d17-13fe-4305-88eb-77c3636749cd-proxy-tls") pod "isvc-canary-create-predictor-854596548c-pv7tx" (UID: "31dd0d17-13fe-4305-88eb-77c3636749cd") : secret "isvc-canary-create-predictor-serving-cert" not found
Sep 09 21:13:19.330321 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:13:19.330044    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dbcd4317-6eac-4497-aa10-5a2867fb1197-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-5cd44d6886-s2gkb\" (UID: \"dbcd4317-6eac-4497-aa10-5a2867fb1197\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5cd44d6886-s2gkb"
Sep 09 21:13:19.330321 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:13:19.330102    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/31dd0d17-13fe-4305-88eb-77c3636749cd-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-854596548c-pv7tx\" (UID: \"31dd0d17-13fe-4305-88eb-77c3636749cd\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-pv7tx"
Sep 09 21:13:19.330321 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:13:19.330133    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/31dd0d17-13fe-4305-88eb-77c3636749cd-kserve-provision-location\") pod \"isvc-canary-create-predictor-854596548c-pv7tx\" (UID: \"31dd0d17-13fe-4305-88eb-77c3636749cd\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-pv7tx"
Sep 09 21:13:19.330321 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:13:19.330169    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/31dd0d17-13fe-4305-88eb-77c3636749cd-cabundle-cert\") pod \"isvc-canary-create-predictor-854596548c-pv7tx\" (UID: \"31dd0d17-13fe-4305-88eb-77c3636749cd\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-pv7tx"
Sep 09 21:13:19.330321 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:13:19.330204    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dbcd4317-6eac-4497-aa10-5a2867fb1197-proxy-tls\") pod \"isvc-canary-create-v2-predictor-5cd44d6886-s2gkb\" (UID: \"dbcd4317-6eac-4497-aa10-5a2867fb1197\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5cd44d6886-s2gkb"
Sep 09 21:13:19.330871 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:13:19.330474    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dbcd4317-6eac-4497-aa10-5a2867fb1197-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-5cd44d6886-s2gkb\" (UID: \"dbcd4317-6eac-4497-aa10-5a2867fb1197\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5cd44d6886-s2gkb"
Sep 09 21:13:19.330871 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:13:19.330648    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/31dd0d17-13fe-4305-88eb-77c3636749cd-kserve-provision-location\") pod \"isvc-canary-create-predictor-854596548c-pv7tx\" (UID: \"31dd0d17-13fe-4305-88eb-77c3636749cd\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-pv7tx"
Sep 09 21:13:19.330871 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:13:19.330666    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dbcd4317-6eac-4497-aa10-5a2867fb1197-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-5cd44d6886-s2gkb\" (UID: \"dbcd4317-6eac-4497-aa10-5a2867fb1197\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5cd44d6886-s2gkb"
Sep 09 21:13:19.330871 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:13:19.330694    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dbcd4317-6eac-4497-aa10-5a2867fb1197-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-5cd44d6886-s2gkb\" (UID: \"dbcd4317-6eac-4497-aa10-5a2867fb1197\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5cd44d6886-s2gkb"
Sep 09 21:13:19.330871 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:13:19.330856    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/31dd0d17-13fe-4305-88eb-77c3636749cd-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-854596548c-pv7tx\" (UID: \"31dd0d17-13fe-4305-88eb-77c3636749cd\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-pv7tx"
Sep 09 21:13:19.331091 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:13:19.331072    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/31dd0d17-13fe-4305-88eb-77c3636749cd-cabundle-cert\") pod \"isvc-canary-create-predictor-854596548c-pv7tx\" (UID: \"31dd0d17-13fe-4305-88eb-77c3636749cd\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-pv7tx"
Sep 09 21:13:19.332800 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:13:19.332774    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dbcd4317-6eac-4497-aa10-5a2867fb1197-proxy-tls\") pod \"isvc-canary-create-v2-predictor-5cd44d6886-s2gkb\" (UID: \"dbcd4317-6eac-4497-aa10-5a2867fb1197\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5cd44d6886-s2gkb"
Sep 09 21:13:19.339464 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:13:19.339420    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnj9r\" (UniqueName: \"kubernetes.io/projected/dbcd4317-6eac-4497-aa10-5a2867fb1197-kube-api-access-lnj9r\") pod \"isvc-canary-create-v2-predictor-5cd44d6886-s2gkb\" (UID: \"dbcd4317-6eac-4497-aa10-5a2867fb1197\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5cd44d6886-s2gkb"
Sep 09 21:13:19.339764 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:13:19.339743    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7ssv\" (UniqueName: \"kubernetes.io/projected/31dd0d17-13fe-4305-88eb-77c3636749cd-kube-api-access-w7ssv\") pod \"isvc-canary-create-predictor-854596548c-pv7tx\" (UID: \"31dd0d17-13fe-4305-88eb-77c3636749cd\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-pv7tx"
Sep 09 21:13:19.421468 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:13:19.421422    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5cd44d6886-s2gkb"
Sep 09 21:13:19.699942 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:13:19.699887    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5cd44d6886-s2gkb"]
Sep 09 21:13:19.702182 ip-10-0-138-224 kubenswrapper[2579]: W0909 21:13:19.702156    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbcd4317_6eac_4497_aa10_5a2867fb1197.slice/crio-10ec1096b959e27893477d7d48445b0234859a19abe344be39f590d028b48718 WatchSource:0}: Error finding container 10ec1096b959e27893477d7d48445b0234859a19abe344be39f590d028b48718: Status 404 returned error can't find the container with id 10ec1096b959e27893477d7d48445b0234859a19abe344be39f590d028b48718
Sep 09 21:13:19.834368 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:13:19.834335    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/31dd0d17-13fe-4305-88eb-77c3636749cd-proxy-tls\") pod \"isvc-canary-create-predictor-854596548c-pv7tx\" (UID: \"31dd0d17-13fe-4305-88eb-77c3636749cd\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-pv7tx"
Sep 09 21:13:19.836748 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:13:19.836724    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/31dd0d17-13fe-4305-88eb-77c3636749cd-proxy-tls\") pod \"isvc-canary-create-predictor-854596548c-pv7tx\" (UID: \"31dd0d17-13fe-4305-88eb-77c3636749cd\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-pv7tx"
Sep 09 21:13:20.018934 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:13:20.018841    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5cd44d6886-s2gkb" event={"ID":"dbcd4317-6eac-4497-aa10-5a2867fb1197","Type":"ContainerStarted","Data":"1093a270518fd2df31c274dd8407048ecb4f03a39eabd86044353d84a550bf08"}
Sep 09 21:13:20.018934 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:13:20.018906    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5cd44d6886-s2gkb" event={"ID":"dbcd4317-6eac-4497-aa10-5a2867fb1197","Type":"ContainerStarted","Data":"10ec1096b959e27893477d7d48445b0234859a19abe344be39f590d028b48718"}
Sep 09 21:13:20.069602 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:13:20.069567    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-pv7tx"
Sep 09 21:13:20.352032 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:13:20.349059    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-pv7tx"]
Sep 09 21:13:20.356121 ip-10-0-138-224 kubenswrapper[2579]: W0909 21:13:20.356090    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31dd0d17_13fe_4305_88eb_77c3636749cd.slice/crio-aac1c2c2d2b8311fa9c61d59386550568d2ec00f1c6b5b6d0ec68c5300041469 WatchSource:0}: Error finding container aac1c2c2d2b8311fa9c61d59386550568d2ec00f1c6b5b6d0ec68c5300041469: Status 404 returned error can't find the container with id aac1c2c2d2b8311fa9c61d59386550568d2ec00f1c6b5b6d0ec68c5300041469
Sep 09 21:13:21.024930 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:13:21.024887    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-pv7tx" event={"ID":"31dd0d17-13fe-4305-88eb-77c3636749cd","Type":"ContainerStarted","Data":"f940f6c5dc913c0ad4d07aca394040babe14d2ad919ef3043c9aff647fc62669"}
Sep 09 21:13:21.025116 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:13:21.024950    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-pv7tx" event={"ID":"31dd0d17-13fe-4305-88eb-77c3636749cd","Type":"ContainerStarted","Data":"aac1c2c2d2b8311fa9c61d59386550568d2ec00f1c6b5b6d0ec68c5300041469"}
Sep 09 21:13:22.270551 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:13:22.270508    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-762cd-predictor-6b8d96d748-lw8dj" podUID="d0c5c989-6102-415f-9bfc-faeb8eec7311" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.41:8643/healthz\": dial tcp 10.134.0.41:8643: connect: connection refused"
Sep 09 21:13:23.762528 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:13:23.762470    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-762cd-predictor-6b8d96d748-lw8dj"
Sep 09 21:13:23.869224 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:13:23.869192    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d0c5c989-6102-415f-9bfc-faeb8eec7311-kube-rbac-proxy-sar-config\") pod \"d0c5c989-6102-415f-9bfc-faeb8eec7311\" (UID: \"d0c5c989-6102-415f-9bfc-faeb8eec7311\") "
Sep 09 21:13:23.869395 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:13:23.869245    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d0c5c989-6102-415f-9bfc-faeb8eec7311-kserve-provision-location\") pod \"d0c5c989-6102-415f-9bfc-faeb8eec7311\" (UID: \"d0c5c989-6102-415f-9bfc-faeb8eec7311\") "
Sep 09 21:13:23.869395 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:13:23.869289    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d0c5c989-6102-415f-9bfc-faeb8eec7311-proxy-tls\") pod \"d0c5c989-6102-415f-9bfc-faeb8eec7311\" (UID: \"d0c5c989-6102-415f-9bfc-faeb8eec7311\") "
Sep 09 21:13:23.869395 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:13:23.869316    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d0c5c989-6102-415f-9bfc-faeb8eec7311-cabundle-cert\") pod \"d0c5c989-6102-415f-9bfc-faeb8eec7311\" (UID: \"d0c5c989-6102-415f-9bfc-faeb8eec7311\") "
Sep 09 21:13:23.869395 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:13:23.869350    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-79w4j\" (UniqueName: \"kubernetes.io/projected/d0c5c989-6102-415f-9bfc-faeb8eec7311-kube-api-access-79w4j\") pod \"d0c5c989-6102-415f-9bfc-faeb8eec7311\" (UID: \"d0c5c989-6102-415f-9bfc-faeb8eec7311\") "
Sep 09 21:13:23.869707 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:13:23.869683    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0c5c989-6102-415f-9bfc-faeb8eec7311-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d0c5c989-6102-415f-9bfc-faeb8eec7311" (UID: "d0c5c989-6102-415f-9bfc-faeb8eec7311"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:13:23.869768 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:13:23.869698    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0c5c989-6102-415f-9bfc-faeb8eec7311-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d0c5c989-6102-415f-9bfc-faeb8eec7311" (UID: "d0c5c989-6102-415f-9bfc-faeb8eec7311"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:13:23.869768 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:13:23.869741    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0c5c989-6102-415f-9bfc-faeb8eec7311-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d0c5c989-6102-415f-9bfc-faeb8eec7311" (UID: "d0c5c989-6102-415f-9bfc-faeb8eec7311"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:13:23.871619 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:13:23.871594    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0c5c989-6102-415f-9bfc-faeb8eec7311-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d0c5c989-6102-415f-9bfc-faeb8eec7311" (UID: "d0c5c989-6102-415f-9bfc-faeb8eec7311"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:13:23.871948 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:13:23.871928    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0c5c989-6102-415f-9bfc-faeb8eec7311-kube-api-access-79w4j" (OuterVolumeSpecName: "kube-api-access-79w4j") pod "d0c5c989-6102-415f-9bfc-faeb8eec7311" (UID: "d0c5c989-6102-415f-9bfc-faeb8eec7311"). InnerVolumeSpecName "kube-api-access-79w4j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:13:23.970800 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:13:23.970753    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-79w4j\" (UniqueName: \"kubernetes.io/projected/d0c5c989-6102-415f-9bfc-faeb8eec7311-kube-api-access-79w4j\") on node \"ip-10-0-138-224.ec2.internal\" DevicePath \"\""
Sep 09 21:13:23.970800 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:13:23.970796    2579 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d0c5c989-6102-415f-9bfc-faeb8eec7311-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-224.ec2.internal\" DevicePath \"\""
Sep 09 21:13:23.970800 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:13:23.970809    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d0c5c989-6102-415f-9bfc-faeb8eec7311-kserve-provision-location\") on node \"ip-10-0-138-224.ec2.internal\" DevicePath \"\""
Sep 09 21:13:23.971023 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:13:23.970826    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d0c5c989-6102-415f-9bfc-faeb8eec7311-proxy-tls\") on node \"ip-10-0-138-224.ec2.internal\" DevicePath \"\""
Sep 09 21:13:23.971023 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:13:23.970839    2579 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d0c5c989-6102-415f-9bfc-faeb8eec7311-cabundle-cert\") on node \"ip-10-0-138-224.ec2.internal\" DevicePath \"\""
Sep 09 21:13:24.039998 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:13:24.039971    2579 generic.go:356] "Generic (PLEG): container finished" podID="dbcd4317-6eac-4497-aa10-5a2867fb1197" containerID="1093a270518fd2df31c274dd8407048ecb4f03a39eabd86044353d84a550bf08" exitCode=0
Sep 09 21:13:24.040118 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:13:24.040038    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5cd44d6886-s2gkb" event={"ID":"dbcd4317-6eac-4497-aa10-5a2867fb1197","Type":"ContainerDied","Data":"1093a270518fd2df31c274dd8407048ecb4f03a39eabd86044353d84a550bf08"}
Sep 09 21:13:24.042059 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:13:24.042037    2579 generic.go:356] "Generic (PLEG): container finished" podID="d0c5c989-6102-415f-9bfc-faeb8eec7311" containerID="67ce1ba30caea136f1add0232d38643138a6e3073856a83ce3fe087fda05da85" exitCode=0
Sep 09 21:13:24.042154 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:13:24.042091    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-762cd-predictor-6b8d96d748-lw8dj"
Sep 09 21:13:24.042154 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:13:24.042116    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-762cd-predictor-6b8d96d748-lw8dj" event={"ID":"d0c5c989-6102-415f-9bfc-faeb8eec7311","Type":"ContainerDied","Data":"67ce1ba30caea136f1add0232d38643138a6e3073856a83ce3fe087fda05da85"}
Sep 09 21:13:24.042252 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:13:24.042173    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-762cd-predictor-6b8d96d748-lw8dj" event={"ID":"d0c5c989-6102-415f-9bfc-faeb8eec7311","Type":"ContainerDied","Data":"2e817adbd668174a1ddea8d71e04e4c006aa65fe9fa385596e6591a71ba03d58"}
Sep 09 21:13:24.042252 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:13:24.042211    2579 scope.go:117] "RemoveContainer" containerID="adc2d4842367b89720894d97807de9a06fb82041c3a9b1dcecce01e3a4f87271"
Sep 09 21:13:24.043852 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:13:24.043831    2579 generic.go:356] "Generic (PLEG): container finished" podID="31dd0d17-13fe-4305-88eb-77c3636749cd" containerID="f940f6c5dc913c0ad4d07aca394040babe14d2ad919ef3043c9aff647fc62669" exitCode=0
Sep 09 21:13:24.043952 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:13:24.043872    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-pv7tx" event={"ID":"31dd0d17-13fe-4305-88eb-77c3636749cd","Type":"ContainerDied","Data":"f940f6c5dc913c0ad4d07aca394040babe14d2ad919ef3043c9aff647fc62669"}
Sep 09 21:13:24.081519 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:13:24.081482    2579 scope.go:117] "RemoveContainer" containerID="67ce1ba30caea136f1add0232d38643138a6e3073856a83ce3fe087fda05da85"
Sep 09 21:13:24.099924 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:13:24.099902    2579 scope.go:117] "RemoveContainer" containerID="5cb094094252f4ee991eb2e0862e2f1ba7019dcd049cab8ca0a665c8d1f65583"
Sep 09 21:13:24.111766 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:13:24.111747    2579 scope.go:117] "RemoveContainer" containerID="adc2d4842367b89720894d97807de9a06fb82041c3a9b1dcecce01e3a4f87271"
Sep 09 21:13:24.112122 ip-10-0-138-224 kubenswrapper[2579]: E0909 21:13:24.112083    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"adc2d4842367b89720894d97807de9a06fb82041c3a9b1dcecce01e3a4f87271\": container with ID starting with adc2d4842367b89720894d97807de9a06fb82041c3a9b1dcecce01e3a4f87271 not found: ID does not exist" containerID="adc2d4842367b89720894d97807de9a06fb82041c3a9b1dcecce01e3a4f87271"
Sep 09 21:13:24.112219 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:13:24.112123    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"adc2d4842367b89720894d97807de9a06fb82041c3a9b1dcecce01e3a4f87271"} err="failed to get container status \"adc2d4842367b89720894d97807de9a06fb82041c3a9b1dcecce01e3a4f87271\": rpc error: code = NotFound desc = could not find container \"adc2d4842367b89720894d97807de9a06fb82041c3a9b1dcecce01e3a4f87271\": container with ID starting with adc2d4842367b89720894d97807de9a06fb82041c3a9b1dcecce01e3a4f87271 not found: ID does not exist"
Sep 09 21:13:24.112219 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:13:24.112146    2579 scope.go:117] "RemoveContainer" containerID="67ce1ba30caea136f1add0232d38643138a6e3073856a83ce3fe087fda05da85"
Sep 09 21:13:24.112602 ip-10-0-138-224 kubenswrapper[2579]: E0909 21:13:24.112580    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67ce1ba30caea136f1add0232d38643138a6e3073856a83ce3fe087fda05da85\": container with ID starting with 67ce1ba30caea136f1add0232d38643138a6e3073856a83ce3fe087fda05da85 not found: ID does not exist" containerID="67ce1ba30caea136f1add0232d38643138a6e3073856a83ce3fe087fda05da85"
Sep 09 21:13:24.112698 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:13:24.112608    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67ce1ba30caea136f1add0232d38643138a6e3073856a83ce3fe087fda05da85"} err="failed to get container status \"67ce1ba30caea136f1add0232d38643138a6e3073856a83ce3fe087fda05da85\": rpc error: code = NotFound desc = could not find container \"67ce1ba30caea136f1add0232d38643138a6e3073856a83ce3fe087fda05da85\": container with ID starting with 67ce1ba30caea136f1add0232d38643138a6e3073856a83ce3fe087fda05da85 not found: ID does not exist"
Sep 09 21:13:24.112698 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:13:24.112623    2579 scope.go:117] "RemoveContainer" containerID="5cb094094252f4ee991eb2e0862e2f1ba7019dcd049cab8ca0a665c8d1f65583"
Sep 09 21:13:24.112909 ip-10-0-138-224 kubenswrapper[2579]: E0909 21:13:24.112888    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5cb094094252f4ee991eb2e0862e2f1ba7019dcd049cab8ca0a665c8d1f65583\": container with ID starting with 5cb094094252f4ee991eb2e0862e2f1ba7019dcd049cab8ca0a665c8d1f65583 not found: ID does not exist" containerID="5cb094094252f4ee991eb2e0862e2f1ba7019dcd049cab8ca0a665c8d1f65583"
Sep 09 21:13:24.112997 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:13:24.112912    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5cb094094252f4ee991eb2e0862e2f1ba7019dcd049cab8ca0a665c8d1f65583"} err="failed to get container status \"5cb094094252f4ee991eb2e0862e2f1ba7019dcd049cab8ca0a665c8d1f65583\": rpc error: code = NotFound desc = could not find container \"5cb094094252f4ee991eb2e0862e2f1ba7019dcd049cab8ca0a665c8d1f65583\": container with ID starting with 5cb094094252f4ee991eb2e0862e2f1ba7019dcd049cab8ca0a665c8d1f65583 not found: ID does not exist"
Sep 09 21:13:24.124426 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:13:24.124402    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-762cd-predictor-6b8d96d748-lw8dj"]
Sep 09 21:13:24.139766 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:13:24.139742    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-762cd-predictor-6b8d96d748-lw8dj"]
Sep 09 21:13:25.051727 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:13:25.051683    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-pv7tx" event={"ID":"31dd0d17-13fe-4305-88eb-77c3636749cd","Type":"ContainerStarted","Data":"560503f5e340aaecc5cf0a5db8474204b1e3e5c39c8482014b8e2728db7a20e4"}
Sep 09 21:13:25.052205 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:13:25.051803    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-pv7tx" event={"ID":"31dd0d17-13fe-4305-88eb-77c3636749cd","Type":"ContainerStarted","Data":"cd371da8e4e4ede78aca762c10fdfb6147dab527e3ed933af05ccc7ef5ce0c7a"}
Sep 09 21:13:25.052205 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:13:25.052127    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-pv7tx"
Sep 09 21:13:25.052339 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:13:25.052313    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-pv7tx"
Sep 09 21:13:25.053883 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:13:25.053856    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-pv7tx" podUID="31dd0d17-13fe-4305-88eb-77c3636749cd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Sep 09 21:13:25.054248 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:13:25.054222    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5cd44d6886-s2gkb" event={"ID":"dbcd4317-6eac-4497-aa10-5a2867fb1197","Type":"ContainerStarted","Data":"3256774bda21d424010d115d4cf5b014360d42cc8ba08a6bd9305de21f2845a8"}
Sep 09 21:13:25.054332 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:13:25.054280    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5cd44d6886-s2gkb" event={"ID":"dbcd4317-6eac-4497-aa10-5a2867fb1197","Type":"ContainerStarted","Data":"b5f316d6fc0067e136c9174d187cba147cdccf003661e9bf70176f44160dcb50"}
Sep 09 21:13:25.054493 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:13:25.054477    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5cd44d6886-s2gkb"
Sep 09 21:13:25.075002 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:13:25.074963    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-pv7tx" podStartSLOduration=2.471096717 podStartE2EDuration="6.074949179s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.603852462s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:13:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:13:25.073108769 +0000 UTC m=+1262.313202898" watchObservedRunningTime="2026-09-09 21:13:25.074949179 +0000 UTC m=+1262.315043297"
Sep 09 21:13:25.097800 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:13:25.097756    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5cd44d6886-s2gkb" podStartSLOduration=2.369033067 podStartE2EDuration="6.097743081s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.728710014s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:13:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:13:25.096723673 +0000 UTC m=+1262.336817792" watchObservedRunningTime="2026-09-09 21:13:25.097743081 +0000 UTC m=+1262.337837198"
Sep 09 21:13:25.293225 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:13:25.293189    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d0c5c989-6102-415f-9bfc-faeb8eec7311" path="/var/lib/kubelet/pods/d0c5c989-6102-415f-9bfc-faeb8eec7311/volumes"
Sep 09 21:13:26.060869 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:13:26.060828    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-pv7tx" podUID="31dd0d17-13fe-4305-88eb-77c3636749cd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Sep 09 21:13:26.061314 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:13:26.060968    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5cd44d6886-s2gkb"
Sep 09 21:13:26.062376 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:13:26.062353    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5cd44d6886-s2gkb" podUID="dbcd4317-6eac-4497-aa10-5a2867fb1197" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Sep 09 21:13:27.065710 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:13:27.065676    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5cd44d6886-s2gkb" podUID="dbcd4317-6eac-4497-aa10-5a2867fb1197" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Sep 09 21:13:31.065434 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:13:31.065406    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-pv7tx"
Sep 09 21:13:31.065960 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:13:31.065933    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-pv7tx" podUID="31dd0d17-13fe-4305-88eb-77c3636749cd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Sep 09 21:13:32.070554 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:13:32.070526    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5cd44d6886-s2gkb"
Sep 09 21:13:32.071014 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:13:32.070989    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5cd44d6886-s2gkb" podUID="dbcd4317-6eac-4497-aa10-5a2867fb1197" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Sep 09 21:13:41.066029 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:13:41.065985    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-pv7tx" podUID="31dd0d17-13fe-4305-88eb-77c3636749cd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Sep 09 21:13:42.071142 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:13:42.071098    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5cd44d6886-s2gkb" podUID="dbcd4317-6eac-4497-aa10-5a2867fb1197" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Sep 09 21:13:51.066747 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:13:51.066702    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-pv7tx" podUID="31dd0d17-13fe-4305-88eb-77c3636749cd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Sep 09 21:13:52.071510 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:13:52.071470    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5cd44d6886-s2gkb" podUID="dbcd4317-6eac-4497-aa10-5a2867fb1197" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Sep 09 21:14:01.066662 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:14:01.066621    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-pv7tx" podUID="31dd0d17-13fe-4305-88eb-77c3636749cd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Sep 09 21:14:02.071826 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:14:02.071788    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5cd44d6886-s2gkb" podUID="dbcd4317-6eac-4497-aa10-5a2867fb1197" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Sep 09 21:14:11.066244 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:14:11.066202    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-pv7tx" podUID="31dd0d17-13fe-4305-88eb-77c3636749cd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Sep 09 21:14:12.071547 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:14:12.071505    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5cd44d6886-s2gkb" podUID="dbcd4317-6eac-4497-aa10-5a2867fb1197" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Sep 09 21:14:21.066691 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:14:21.066650    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-pv7tx" podUID="31dd0d17-13fe-4305-88eb-77c3636749cd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Sep 09 21:14:22.071589 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:14:22.071546    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5cd44d6886-s2gkb" podUID="dbcd4317-6eac-4497-aa10-5a2867fb1197" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Sep 09 21:14:31.065938 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:14:31.065900    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-pv7tx" podUID="31dd0d17-13fe-4305-88eb-77c3636749cd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Sep 09 21:14:32.071726 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:14:32.071680    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5cd44d6886-s2gkb" podUID="dbcd4317-6eac-4497-aa10-5a2867fb1197" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Sep 09 21:14:40.286491 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:14:40.286432    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-pv7tx"
Sep 09 21:14:41.291273 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:14:41.291244    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5cd44d6886-s2gkb"
Sep 09 21:14:49.206360 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:14:49.206283    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5cd44d6886-s2gkb"]
Sep 09 21:14:49.206888 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:14:49.206525    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5cd44d6886-s2gkb" podUID="dbcd4317-6eac-4497-aa10-5a2867fb1197" containerName="kserve-container" containerID="cri-o://b5f316d6fc0067e136c9174d187cba147cdccf003661e9bf70176f44160dcb50" gracePeriod=30
Sep 09 21:14:49.206888 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:14:49.206562    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5cd44d6886-s2gkb" podUID="dbcd4317-6eac-4497-aa10-5a2867fb1197" containerName="kube-rbac-proxy" containerID="cri-o://3256774bda21d424010d115d4cf5b014360d42cc8ba08a6bd9305de21f2845a8" gracePeriod=30
Sep 09 21:14:49.212964 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:14:49.212941    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-pv7tx"]
Sep 09 21:14:49.213183 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:14:49.213159    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-pv7tx" podUID="31dd0d17-13fe-4305-88eb-77c3636749cd" containerName="kserve-container" containerID="cri-o://cd371da8e4e4ede78aca762c10fdfb6147dab527e3ed933af05ccc7ef5ce0c7a" gracePeriod=30
Sep 09 21:14:49.213279 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:14:49.213230    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-pv7tx" podUID="31dd0d17-13fe-4305-88eb-77c3636749cd" containerName="kube-rbac-proxy" containerID="cri-o://560503f5e340aaecc5cf0a5db8474204b1e3e5c39c8482014b8e2728db7a20e4" gracePeriod=30
Sep 09 21:14:49.434814 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:14:49.434779    2579 generic.go:356] "Generic (PLEG): container finished" podID="31dd0d17-13fe-4305-88eb-77c3636749cd" containerID="560503f5e340aaecc5cf0a5db8474204b1e3e5c39c8482014b8e2728db7a20e4" exitCode=2
Sep 09 21:14:49.434963 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:14:49.434811    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-pv7tx" event={"ID":"31dd0d17-13fe-4305-88eb-77c3636749cd","Type":"ContainerDied","Data":"560503f5e340aaecc5cf0a5db8474204b1e3e5c39c8482014b8e2728db7a20e4"}
Sep 09 21:14:49.437080 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:14:49.437055    2579 generic.go:356] "Generic (PLEG): container finished" podID="dbcd4317-6eac-4497-aa10-5a2867fb1197" containerID="3256774bda21d424010d115d4cf5b014360d42cc8ba08a6bd9305de21f2845a8" exitCode=2
Sep 09 21:14:49.437197 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:14:49.437128    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5cd44d6886-s2gkb" event={"ID":"dbcd4317-6eac-4497-aa10-5a2867fb1197","Type":"ContainerDied","Data":"3256774bda21d424010d115d4cf5b014360d42cc8ba08a6bd9305de21f2845a8"}
Sep 09 21:14:50.286085 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:14:50.286045    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-pv7tx" podUID="31dd0d17-13fe-4305-88eb-77c3636749cd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Sep 09 21:14:51.061607 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:14:51.061565    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-pv7tx" podUID="31dd0d17-13fe-4305-88eb-77c3636749cd" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.43:8643/healthz\": dial tcp 10.134.0.43:8643: connect: connection refused"
Sep 09 21:14:51.286260 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:14:51.286211    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5cd44d6886-s2gkb" podUID="dbcd4317-6eac-4497-aa10-5a2867fb1197" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Sep 09 21:14:52.066724 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:14:52.066683    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5cd44d6886-s2gkb" podUID="dbcd4317-6eac-4497-aa10-5a2867fb1197" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.42:8643/healthz\": dial tcp 10.134.0.42:8643: connect: connection refused"
Sep 09 21:14:53.966672 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:14:53.966645    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-pv7tx"
Sep 09 21:14:54.017534 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:14:54.017506    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w7ssv\" (UniqueName: \"kubernetes.io/projected/31dd0d17-13fe-4305-88eb-77c3636749cd-kube-api-access-w7ssv\") pod \"31dd0d17-13fe-4305-88eb-77c3636749cd\" (UID: \"31dd0d17-13fe-4305-88eb-77c3636749cd\") "
Sep 09 21:14:54.017683 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:14:54.017539    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/31dd0d17-13fe-4305-88eb-77c3636749cd-cabundle-cert\") pod \"31dd0d17-13fe-4305-88eb-77c3636749cd\" (UID: \"31dd0d17-13fe-4305-88eb-77c3636749cd\") "
Sep 09 21:14:54.017683 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:14:54.017564    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/31dd0d17-13fe-4305-88eb-77c3636749cd-kserve-provision-location\") pod \"31dd0d17-13fe-4305-88eb-77c3636749cd\" (UID: \"31dd0d17-13fe-4305-88eb-77c3636749cd\") "
Sep 09 21:14:54.017683 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:14:54.017594    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/31dd0d17-13fe-4305-88eb-77c3636749cd-kube-rbac-proxy-sar-config\") pod \"31dd0d17-13fe-4305-88eb-77c3636749cd\" (UID: \"31dd0d17-13fe-4305-88eb-77c3636749cd\") "
Sep 09 21:14:54.017683 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:14:54.017630    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/31dd0d17-13fe-4305-88eb-77c3636749cd-proxy-tls\") pod \"31dd0d17-13fe-4305-88eb-77c3636749cd\" (UID: \"31dd0d17-13fe-4305-88eb-77c3636749cd\") "
Sep 09 21:14:54.017922 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:14:54.017894    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31dd0d17-13fe-4305-88eb-77c3636749cd-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "31dd0d17-13fe-4305-88eb-77c3636749cd" (UID: "31dd0d17-13fe-4305-88eb-77c3636749cd"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:14:54.017992 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:14:54.017938    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31dd0d17-13fe-4305-88eb-77c3636749cd-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "31dd0d17-13fe-4305-88eb-77c3636749cd" (UID: "31dd0d17-13fe-4305-88eb-77c3636749cd"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:14:54.018064 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:14:54.018040    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31dd0d17-13fe-4305-88eb-77c3636749cd-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "31dd0d17-13fe-4305-88eb-77c3636749cd" (UID: "31dd0d17-13fe-4305-88eb-77c3636749cd"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:14:54.021481 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:14:54.021460    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31dd0d17-13fe-4305-88eb-77c3636749cd-kube-api-access-w7ssv" (OuterVolumeSpecName: "kube-api-access-w7ssv") pod "31dd0d17-13fe-4305-88eb-77c3636749cd" (UID: "31dd0d17-13fe-4305-88eb-77c3636749cd"). InnerVolumeSpecName "kube-api-access-w7ssv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:14:54.023400 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:14:54.023382    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31dd0d17-13fe-4305-88eb-77c3636749cd-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "31dd0d17-13fe-4305-88eb-77c3636749cd" (UID: "31dd0d17-13fe-4305-88eb-77c3636749cd"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:14:54.041378 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:14:54.041356    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5cd44d6886-s2gkb"
Sep 09 21:14:54.118683 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:14:54.118652    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dbcd4317-6eac-4497-aa10-5a2867fb1197-proxy-tls\") pod \"dbcd4317-6eac-4497-aa10-5a2867fb1197\" (UID: \"dbcd4317-6eac-4497-aa10-5a2867fb1197\") "
Sep 09 21:14:54.118683 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:14:54.118686    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lnj9r\" (UniqueName: \"kubernetes.io/projected/dbcd4317-6eac-4497-aa10-5a2867fb1197-kube-api-access-lnj9r\") pod \"dbcd4317-6eac-4497-aa10-5a2867fb1197\" (UID: \"dbcd4317-6eac-4497-aa10-5a2867fb1197\") "
Sep 09 21:14:54.118876 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:14:54.118715    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dbcd4317-6eac-4497-aa10-5a2867fb1197-kube-rbac-proxy-sar-config\") pod \"dbcd4317-6eac-4497-aa10-5a2867fb1197\" (UID: \"dbcd4317-6eac-4497-aa10-5a2867fb1197\") "
Sep 09 21:14:54.118876 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:14:54.118739    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dbcd4317-6eac-4497-aa10-5a2867fb1197-cabundle-cert\") pod \"dbcd4317-6eac-4497-aa10-5a2867fb1197\" (UID: \"dbcd4317-6eac-4497-aa10-5a2867fb1197\") "
Sep 09 21:14:54.118876 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:14:54.118768    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dbcd4317-6eac-4497-aa10-5a2867fb1197-kserve-provision-location\") pod \"dbcd4317-6eac-4497-aa10-5a2867fb1197\" (UID: \"dbcd4317-6eac-4497-aa10-5a2867fb1197\") "
Sep 09 21:14:54.119037 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:14:54.118947    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w7ssv\" (UniqueName: \"kubernetes.io/projected/31dd0d17-13fe-4305-88eb-77c3636749cd-kube-api-access-w7ssv\") on node \"ip-10-0-138-224.ec2.internal\" DevicePath \"\""
Sep 09 21:14:54.119037 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:14:54.118966    2579 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/31dd0d17-13fe-4305-88eb-77c3636749cd-cabundle-cert\") on node \"ip-10-0-138-224.ec2.internal\" DevicePath \"\""
Sep 09 21:14:54.119037 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:14:54.118981    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/31dd0d17-13fe-4305-88eb-77c3636749cd-kserve-provision-location\") on node \"ip-10-0-138-224.ec2.internal\" DevicePath \"\""
Sep 09 21:14:54.119037 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:14:54.118994    2579 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/31dd0d17-13fe-4305-88eb-77c3636749cd-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-224.ec2.internal\" DevicePath \"\""
Sep 09 21:14:54.119037 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:14:54.119008    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/31dd0d17-13fe-4305-88eb-77c3636749cd-proxy-tls\") on node \"ip-10-0-138-224.ec2.internal\" DevicePath \"\""
Sep 09 21:14:54.119281 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:14:54.119107    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbcd4317-6eac-4497-aa10-5a2867fb1197-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "dbcd4317-6eac-4497-aa10-5a2867fb1197" (UID: "dbcd4317-6eac-4497-aa10-5a2867fb1197"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:14:54.119281 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:14:54.119128    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbcd4317-6eac-4497-aa10-5a2867fb1197-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "dbcd4317-6eac-4497-aa10-5a2867fb1197" (UID: "dbcd4317-6eac-4497-aa10-5a2867fb1197"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:14:54.119281 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:14:54.119159    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbcd4317-6eac-4497-aa10-5a2867fb1197-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "dbcd4317-6eac-4497-aa10-5a2867fb1197" (UID: "dbcd4317-6eac-4497-aa10-5a2867fb1197"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:14:54.120832 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:14:54.120814    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dbcd4317-6eac-4497-aa10-5a2867fb1197-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "dbcd4317-6eac-4497-aa10-5a2867fb1197" (UID: "dbcd4317-6eac-4497-aa10-5a2867fb1197"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:14:54.120913 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:14:54.120895    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbcd4317-6eac-4497-aa10-5a2867fb1197-kube-api-access-lnj9r" (OuterVolumeSpecName: "kube-api-access-lnj9r") pod "dbcd4317-6eac-4497-aa10-5a2867fb1197" (UID: "dbcd4317-6eac-4497-aa10-5a2867fb1197"). InnerVolumeSpecName "kube-api-access-lnj9r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:14:54.219612 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:14:54.219586    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dbcd4317-6eac-4497-aa10-5a2867fb1197-kserve-provision-location\") on node \"ip-10-0-138-224.ec2.internal\" DevicePath \"\""
Sep 09 21:14:54.219612 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:14:54.219609    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dbcd4317-6eac-4497-aa10-5a2867fb1197-proxy-tls\") on node \"ip-10-0-138-224.ec2.internal\" DevicePath \"\""
Sep 09 21:14:54.219773 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:14:54.219620    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lnj9r\" (UniqueName: \"kubernetes.io/projected/dbcd4317-6eac-4497-aa10-5a2867fb1197-kube-api-access-lnj9r\") on node \"ip-10-0-138-224.ec2.internal\" DevicePath \"\""
Sep 09 21:14:54.219773 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:14:54.219628    2579 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dbcd4317-6eac-4497-aa10-5a2867fb1197-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-224.ec2.internal\" DevicePath \"\""
Sep 09 21:14:54.219773 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:14:54.219636    2579 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dbcd4317-6eac-4497-aa10-5a2867fb1197-cabundle-cert\") on node \"ip-10-0-138-224.ec2.internal\" DevicePath \"\""
Sep 09 21:14:54.387073 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:14:54.387003    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-zf64p"]
Sep 09 21:14:54.387399 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:14:54.387385    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31dd0d17-13fe-4305-88eb-77c3636749cd" containerName="storage-initializer"
Sep 09 21:14:54.387466 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:14:54.387402    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="31dd0d17-13fe-4305-88eb-77c3636749cd" containerName="storage-initializer"
Sep 09 21:14:54.387466 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:14:54.387411    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d0c5c989-6102-415f-9bfc-faeb8eec7311" containerName="storage-initializer"
Sep 09 21:14:54.387466 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:14:54.387420    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0c5c989-6102-415f-9bfc-faeb8eec7311" containerName="storage-initializer"
Sep 09 21:14:54.387466 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:14:54.387434    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dbcd4317-6eac-4497-aa10-5a2867fb1197" containerName="kserve-container"
Sep 09 21:14:54.387466 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:14:54.387457    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="dbcd4317-6eac-4497-aa10-5a2867fb1197" containerName="kserve-container"
Sep 09 21:14:54.387625 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:14:54.387471    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dbcd4317-6eac-4497-aa10-5a2867fb1197" containerName="storage-initializer"
Sep 09 21:14:54.387625 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:14:54.387480    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="dbcd4317-6eac-4497-aa10-5a2867fb1197" containerName="storage-initializer"
Sep 09 21:14:54.387625 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:14:54.387493    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31dd0d17-13fe-4305-88eb-77c3636749cd" containerName="kserve-container"
Sep 09 21:14:54.387625 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:14:54.387501    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="31dd0d17-13fe-4305-88eb-77c3636749cd" containerName="kserve-container"
Sep 09 21:14:54.387625 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:14:54.387519    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d0c5c989-6102-415f-9bfc-faeb8eec7311" containerName="kube-rbac-proxy"
Sep 09 21:14:54.387625 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:14:54.387525    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0c5c989-6102-415f-9bfc-faeb8eec7311" containerName="kube-rbac-proxy"
Sep 09 21:14:54.387625 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:14:54.387534    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dbcd4317-6eac-4497-aa10-5a2867fb1197" containerName="kube-rbac-proxy"
Sep 09 21:14:54.387625 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:14:54.387539    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="dbcd4317-6eac-4497-aa10-5a2867fb1197" containerName="kube-rbac-proxy"
Sep 09 21:14:54.387625 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:14:54.387546    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d0c5c989-6102-415f-9bfc-faeb8eec7311" containerName="kserve-container"
Sep 09 21:14:54.387625 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:14:54.387551    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0c5c989-6102-415f-9bfc-faeb8eec7311" containerName="kserve-container"
Sep 09 21:14:54.387625 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:14:54.387585    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31dd0d17-13fe-4305-88eb-77c3636749cd" containerName="kube-rbac-proxy"
Sep 09 21:14:54.387625 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:14:54.387594    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="31dd0d17-13fe-4305-88eb-77c3636749cd" containerName="kube-rbac-proxy"
Sep 09 21:14:54.387974 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:14:54.387677    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="dbcd4317-6eac-4497-aa10-5a2867fb1197" containerName="kube-rbac-proxy"
Sep 09 21:14:54.387974 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:14:54.387689    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="31dd0d17-13fe-4305-88eb-77c3636749cd" containerName="kube-rbac-proxy"
Sep 09 21:14:54.387974 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:14:54.387697    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="dbcd4317-6eac-4497-aa10-5a2867fb1197" containerName="kserve-container"
Sep 09 21:14:54.387974 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:14:54.387710    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="d0c5c989-6102-415f-9bfc-faeb8eec7311" containerName="kserve-container"
Sep 09 21:14:54.387974 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:14:54.387718    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="d0c5c989-6102-415f-9bfc-faeb8eec7311" containerName="kube-rbac-proxy"
Sep 09 21:14:54.387974 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:14:54.387723    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="31dd0d17-13fe-4305-88eb-77c3636749cd" containerName="kserve-container"
Sep 09 21:14:54.391036 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:14:54.391018    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-zf64p"
Sep 09 21:14:54.393554 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:14:54.393530    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-v2-predictor-serving-cert\""
Sep 09 21:14:54.393695 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:14:54.393583    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-kube-rbac-proxy-sar-config\""
Sep 09 21:14:54.413745 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:14:54.413722    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-zf64p"]
Sep 09 21:14:54.421266 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:14:54.421243    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/34726418-cf49-4e02-a3ae-c682369e582a-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-b8964cf76-zf64p\" (UID: \"34726418-cf49-4e02-a3ae-c682369e582a\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-zf64p"
Sep 09 21:14:54.421367 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:14:54.421277    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/34726418-cf49-4e02-a3ae-c682369e582a-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-b8964cf76-zf64p\" (UID: \"34726418-cf49-4e02-a3ae-c682369e582a\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-zf64p"
Sep 09 21:14:54.421367 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:14:54.421310    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x47s2\" (UniqueName: \"kubernetes.io/projected/34726418-cf49-4e02-a3ae-c682369e582a-kube-api-access-x47s2\") pod \"isvc-canary-promote-v2-predictor-b8964cf76-zf64p\" (UID: \"34726418-cf49-4e02-a3ae-c682369e582a\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-zf64p"
Sep 09 21:14:54.421537 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:14:54.421510    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/34726418-cf49-4e02-a3ae-c682369e582a-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-b8964cf76-zf64p\" (UID: \"34726418-cf49-4e02-a3ae-c682369e582a\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-zf64p"
Sep 09 21:14:54.421588 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:14:54.421560    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/34726418-cf49-4e02-a3ae-c682369e582a-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-b8964cf76-zf64p\" (UID: \"34726418-cf49-4e02-a3ae-c682369e582a\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-zf64p"
Sep 09 21:14:54.453949 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:14:54.453917    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-jhzwl"]
Sep 09 21:14:54.457432 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:14:54.457394    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-jhzwl"
Sep 09 21:14:54.459857 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:14:54.459835    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-predictor-serving-cert\""
Sep 09 21:14:54.461896 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:14:54.461869    2579 generic.go:356] "Generic (PLEG): container finished" podID="dbcd4317-6eac-4497-aa10-5a2867fb1197" containerID="b5f316d6fc0067e136c9174d187cba147cdccf003661e9bf70176f44160dcb50" exitCode=0
Sep 09 21:14:54.461971 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:14:54.461941    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5cd44d6886-s2gkb"
Sep 09 21:14:54.462024 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:14:54.462003    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5cd44d6886-s2gkb" event={"ID":"dbcd4317-6eac-4497-aa10-5a2867fb1197","Type":"ContainerDied","Data":"b5f316d6fc0067e136c9174d187cba147cdccf003661e9bf70176f44160dcb50"}
Sep 09 21:14:54.462087 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:14:54.462073    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5cd44d6886-s2gkb" event={"ID":"dbcd4317-6eac-4497-aa10-5a2867fb1197","Type":"ContainerDied","Data":"10ec1096b959e27893477d7d48445b0234859a19abe344be39f590d028b48718"}
Sep 09 21:14:54.462156 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:14:54.462115    2579 scope.go:117] "RemoveContainer" containerID="3256774bda21d424010d115d4cf5b014360d42cc8ba08a6bd9305de21f2845a8"
Sep 09 21:14:54.464379 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:14:54.464351    2579 generic.go:356] "Generic (PLEG): container finished" podID="31dd0d17-13fe-4305-88eb-77c3636749cd" containerID="cd371da8e4e4ede78aca762c10fdfb6147dab527e3ed933af05ccc7ef5ce0c7a" exitCode=0
Sep 09 21:14:54.464494 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:14:54.464405    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-pv7tx" event={"ID":"31dd0d17-13fe-4305-88eb-77c3636749cd","Type":"ContainerDied","Data":"cd371da8e4e4ede78aca762c10fdfb6147dab527e3ed933af05ccc7ef5ce0c7a"}
Sep 09 21:14:54.464494 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:14:54.464414    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-pv7tx"
Sep 09 21:14:54.464494 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:14:54.464475    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-pv7tx" event={"ID":"31dd0d17-13fe-4305-88eb-77c3636749cd","Type":"ContainerDied","Data":"aac1c2c2d2b8311fa9c61d59386550568d2ec00f1c6b5b6d0ec68c5300041469"}
Sep 09 21:14:54.469770 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:14:54.469748    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-jhzwl"]
Sep 09 21:14:54.478781 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:14:54.477229    2579 scope.go:117] "RemoveContainer" containerID="b5f316d6fc0067e136c9174d187cba147cdccf003661e9bf70176f44160dcb50"
Sep 09 21:14:54.486950 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:14:54.486930    2579 scope.go:117] "RemoveContainer" containerID="1093a270518fd2df31c274dd8407048ecb4f03a39eabd86044353d84a550bf08"
Sep 09 21:14:54.495216 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:14:54.495197    2579 scope.go:117] "RemoveContainer" containerID="3256774bda21d424010d115d4cf5b014360d42cc8ba08a6bd9305de21f2845a8"
Sep 09 21:14:54.495539 ip-10-0-138-224 kubenswrapper[2579]: E0909 21:14:54.495511    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3256774bda21d424010d115d4cf5b014360d42cc8ba08a6bd9305de21f2845a8\": container with ID starting with 3256774bda21d424010d115d4cf5b014360d42cc8ba08a6bd9305de21f2845a8 not found: ID does not exist" containerID="3256774bda21d424010d115d4cf5b014360d42cc8ba08a6bd9305de21f2845a8"
Sep 09 21:14:54.495608 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:14:54.495549    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3256774bda21d424010d115d4cf5b014360d42cc8ba08a6bd9305de21f2845a8"} err="failed to get container status \"3256774bda21d424010d115d4cf5b014360d42cc8ba08a6bd9305de21f2845a8\": rpc error: code = NotFound desc = could not find container \"3256774bda21d424010d115d4cf5b014360d42cc8ba08a6bd9305de21f2845a8\": container with ID starting with 3256774bda21d424010d115d4cf5b014360d42cc8ba08a6bd9305de21f2845a8 not found: ID does not exist"
Sep 09 21:14:54.495608 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:14:54.495567    2579 scope.go:117] "RemoveContainer" containerID="b5f316d6fc0067e136c9174d187cba147cdccf003661e9bf70176f44160dcb50"
Sep 09 21:14:54.495815 ip-10-0-138-224 kubenswrapper[2579]: E0909 21:14:54.495800    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5f316d6fc0067e136c9174d187cba147cdccf003661e9bf70176f44160dcb50\": container with ID starting with b5f316d6fc0067e136c9174d187cba147cdccf003661e9bf70176f44160dcb50 not found: ID does not exist" containerID="b5f316d6fc0067e136c9174d187cba147cdccf003661e9bf70176f44160dcb50"
Sep 09 21:14:54.495865 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:14:54.495819    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5f316d6fc0067e136c9174d187cba147cdccf003661e9bf70176f44160dcb50"} err="failed to get container status \"b5f316d6fc0067e136c9174d187cba147cdccf003661e9bf70176f44160dcb50\": rpc error: code = NotFound desc = could not find container \"b5f316d6fc0067e136c9174d187cba147cdccf003661e9bf70176f44160dcb50\": container with ID starting with b5f316d6fc0067e136c9174d187cba147cdccf003661e9bf70176f44160dcb50 not found: ID does not exist"
Sep 09 21:14:54.495865 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:14:54.495830    2579 scope.go:117] "RemoveContainer" containerID="1093a270518fd2df31c274dd8407048ecb4f03a39eabd86044353d84a550bf08"
Sep 09 21:14:54.496095 ip-10-0-138-224 kubenswrapper[2579]: E0909 21:14:54.496079    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1093a270518fd2df31c274dd8407048ecb4f03a39eabd86044353d84a550bf08\": container with ID starting with 1093a270518fd2df31c274dd8407048ecb4f03a39eabd86044353d84a550bf08 not found: ID does not exist" containerID="1093a270518fd2df31c274dd8407048ecb4f03a39eabd86044353d84a550bf08"
Sep 09 21:14:54.496136 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:14:54.496100    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1093a270518fd2df31c274dd8407048ecb4f03a39eabd86044353d84a550bf08"} err="failed to get container status \"1093a270518fd2df31c274dd8407048ecb4f03a39eabd86044353d84a550bf08\": rpc error: code = NotFound desc = could not find container \"1093a270518fd2df31c274dd8407048ecb4f03a39eabd86044353d84a550bf08\": container with ID starting with 1093a270518fd2df31c274dd8407048ecb4f03a39eabd86044353d84a550bf08 not found: ID does not exist"
Sep 09 21:14:54.496136 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:14:54.496114    2579 scope.go:117] "RemoveContainer" containerID="560503f5e340aaecc5cf0a5db8474204b1e3e5c39c8482014b8e2728db7a20e4"
Sep 09 21:14:54.500674 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:14:54.500655    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5cd44d6886-s2gkb"]
Sep 09 21:14:54.503931 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:14:54.503911    2579 scope.go:117] "RemoveContainer" containerID="cd371da8e4e4ede78aca762c10fdfb6147dab527e3ed933af05ccc7ef5ce0c7a"
Sep 09 21:14:54.507129 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:14:54.507108    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5cd44d6886-s2gkb"]
Sep 09 21:14:54.512512 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:14:54.512449    2579 scope.go:117] "RemoveContainer" containerID="f940f6c5dc913c0ad4d07aca394040babe14d2ad919ef3043c9aff647fc62669"
Sep 09 21:14:54.520476 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:14:54.520431    2579 scope.go:117] "RemoveContainer" containerID="560503f5e340aaecc5cf0a5db8474204b1e3e5c39c8482014b8e2728db7a20e4"
Sep 09 21:14:54.520760 ip-10-0-138-224 kubenswrapper[2579]: E0909 21:14:54.520741    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"560503f5e340aaecc5cf0a5db8474204b1e3e5c39c8482014b8e2728db7a20e4\": container with ID starting with 560503f5e340aaecc5cf0a5db8474204b1e3e5c39c8482014b8e2728db7a20e4 not found: ID does not exist" containerID="560503f5e340aaecc5cf0a5db8474204b1e3e5c39c8482014b8e2728db7a20e4"
Sep 09 21:14:54.520813 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:14:54.520767    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"560503f5e340aaecc5cf0a5db8474204b1e3e5c39c8482014b8e2728db7a20e4"} err="failed to get container status \"560503f5e340aaecc5cf0a5db8474204b1e3e5c39c8482014b8e2728db7a20e4\": rpc error: code = NotFound desc = could not find container \"560503f5e340aaecc5cf0a5db8474204b1e3e5c39c8482014b8e2728db7a20e4\": container with ID starting with 560503f5e340aaecc5cf0a5db8474204b1e3e5c39c8482014b8e2728db7a20e4 not found: ID does not exist"
Sep 09 21:14:54.520813 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:14:54.520785    2579 scope.go:117] "RemoveContainer" containerID="cd371da8e4e4ede78aca762c10fdfb6147dab527e3ed933af05ccc7ef5ce0c7a"
Sep 09 21:14:54.521055 ip-10-0-138-224 kubenswrapper[2579]: E0909 21:14:54.521039    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd371da8e4e4ede78aca762c10fdfb6147dab527e3ed933af05ccc7ef5ce0c7a\": container with ID starting with cd371da8e4e4ede78aca762c10fdfb6147dab527e3ed933af05ccc7ef5ce0c7a not found: ID does not exist" containerID="cd371da8e4e4ede78aca762c10fdfb6147dab527e3ed933af05ccc7ef5ce0c7a"
Sep 09 21:14:54.521098 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:14:54.521061    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd371da8e4e4ede78aca762c10fdfb6147dab527e3ed933af05ccc7ef5ce0c7a"} err="failed to get container status \"cd371da8e4e4ede78aca762c10fdfb6147dab527e3ed933af05ccc7ef5ce0c7a\": rpc error: code = NotFound desc = could not find container \"cd371da8e4e4ede78aca762c10fdfb6147dab527e3ed933af05ccc7ef5ce0c7a\": container with ID starting with cd371da8e4e4ede78aca762c10fdfb6147dab527e3ed933af05ccc7ef5ce0c7a not found: ID does not exist"
Sep 09 21:14:54.521098 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:14:54.521075    2579 scope.go:117] "RemoveContainer" containerID="f940f6c5dc913c0ad4d07aca394040babe14d2ad919ef3043c9aff647fc62669"
Sep 09 21:14:54.521348 ip-10-0-138-224 kubenswrapper[2579]: E0909 21:14:54.521332    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f940f6c5dc913c0ad4d07aca394040babe14d2ad919ef3043c9aff647fc62669\": container with ID starting with f940f6c5dc913c0ad4d07aca394040babe14d2ad919ef3043c9aff647fc62669 not found: ID does not exist" containerID="f940f6c5dc913c0ad4d07aca394040babe14d2ad919ef3043c9aff647fc62669"
Sep 09 21:14:54.521396 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:14:54.521352    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f940f6c5dc913c0ad4d07aca394040babe14d2ad919ef3043c9aff647fc62669"} err="failed to get container status \"f940f6c5dc913c0ad4d07aca394040babe14d2ad919ef3043c9aff647fc62669\": rpc error: code = NotFound desc = could not find container \"f940f6c5dc913c0ad4d07aca394040babe14d2ad919ef3043c9aff647fc62669\": container with ID starting with f940f6c5dc913c0ad4d07aca394040babe14d2ad919ef3043c9aff647fc62669 not found: ID does not exist"
Sep 09 21:14:54.522805 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:14:54.522778    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4bcq9\" (UniqueName: \"kubernetes.io/projected/f68bf759-7bee-4078-95f2-38368fae9f0a-kube-api-access-4bcq9\") pod \"isvc-canary-promote-predictor-65ff4fc5f5-jhzwl\" (UID: \"f68bf759-7bee-4078-95f2-38368fae9f0a\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-jhzwl"
Sep 09 21:14:54.522904 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:14:54.522854    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/34726418-cf49-4e02-a3ae-c682369e582a-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-b8964cf76-zf64p\" (UID: \"34726418-cf49-4e02-a3ae-c682369e582a\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-zf64p"
Sep 09 21:14:54.522966 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:14:54.522905    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/34726418-cf49-4e02-a3ae-c682369e582a-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-b8964cf76-zf64p\" (UID: \"34726418-cf49-4e02-a3ae-c682369e582a\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-zf64p"
Sep 09 21:14:54.522966 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:14:54.522926    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f68bf759-7bee-4078-95f2-38368fae9f0a-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-65ff4fc5f5-jhzwl\" (UID: \"f68bf759-7bee-4078-95f2-38368fae9f0a\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-jhzwl"
Sep 09 21:14:54.522966 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:14:54.522955    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/34726418-cf49-4e02-a3ae-c682369e582a-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-b8964cf76-zf64p\" (UID: \"34726418-cf49-4e02-a3ae-c682369e582a\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-zf64p"
Sep 09 21:14:54.523126 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:14:54.523005    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x47s2\" (UniqueName: \"kubernetes.io/projected/34726418-cf49-4e02-a3ae-c682369e582a-kube-api-access-x47s2\") pod \"isvc-canary-promote-v2-predictor-b8964cf76-zf64p\" (UID: \"34726418-cf49-4e02-a3ae-c682369e582a\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-zf64p"
Sep 09 21:14:54.523126 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:14:54.523092    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f68bf759-7bee-4078-95f2-38368fae9f0a-cabundle-cert\") pod \"isvc-canary-promote-predictor-65ff4fc5f5-jhzwl\" (UID: \"f68bf759-7bee-4078-95f2-38368fae9f0a\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-jhzwl"
Sep 09 21:14:54.523233 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:14:54.523134    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f68bf759-7bee-4078-95f2-38368fae9f0a-proxy-tls\") pod \"isvc-canary-promote-predictor-65ff4fc5f5-jhzwl\" (UID: \"f68bf759-7bee-4078-95f2-38368fae9f0a\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-jhzwl"
Sep 09 21:14:54.523233 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:14:54.523167    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f68bf759-7bee-4078-95f2-38368fae9f0a-kserve-provision-location\") pod \"isvc-canary-promote-predictor-65ff4fc5f5-jhzwl\" (UID: \"f68bf759-7bee-4078-95f2-38368fae9f0a\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-jhzwl"
Sep 09 21:14:54.523233 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:14:54.523217    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/34726418-cf49-4e02-a3ae-c682369e582a-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-b8964cf76-zf64p\" (UID: \"34726418-cf49-4e02-a3ae-c682369e582a\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-zf64p"
Sep 09 21:14:54.523579 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:14:54.523550    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/34726418-cf49-4e02-a3ae-c682369e582a-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-b8964cf76-zf64p\" (UID: \"34726418-cf49-4e02-a3ae-c682369e582a\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-zf64p"
Sep 09 21:14:54.523579 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:14:54.523570    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/34726418-cf49-4e02-a3ae-c682369e582a-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-b8964cf76-zf64p\" (UID: \"34726418-cf49-4e02-a3ae-c682369e582a\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-zf64p"
Sep 09 21:14:54.523758 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:14:54.523612    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/34726418-cf49-4e02-a3ae-c682369e582a-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-b8964cf76-zf64p\" (UID: \"34726418-cf49-4e02-a3ae-c682369e582a\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-zf64p"
Sep 09 21:14:54.524667 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:14:54.524650    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-pv7tx"]
Sep 09 21:14:54.525811 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:14:54.525795    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/34726418-cf49-4e02-a3ae-c682369e582a-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-b8964cf76-zf64p\" (UID: \"34726418-cf49-4e02-a3ae-c682369e582a\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-zf64p"
Sep 09 21:14:54.531381 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:14:54.531361    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-pv7tx"]
Sep 09 21:14:54.533504 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:14:54.533484    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x47s2\" (UniqueName: \"kubernetes.io/projected/34726418-cf49-4e02-a3ae-c682369e582a-kube-api-access-x47s2\") pod \"isvc-canary-promote-v2-predictor-b8964cf76-zf64p\" (UID: \"34726418-cf49-4e02-a3ae-c682369e582a\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-zf64p"
Sep 09 21:14:54.624033 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:14:54.624007    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f68bf759-7bee-4078-95f2-38368fae9f0a-proxy-tls\") pod \"isvc-canary-promote-predictor-65ff4fc5f5-jhzwl\" (UID: \"f68bf759-7bee-4078-95f2-38368fae9f0a\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-jhzwl"
Sep 09 21:14:54.624183 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:14:54.624036    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f68bf759-7bee-4078-95f2-38368fae9f0a-kserve-provision-location\") pod \"isvc-canary-promote-predictor-65ff4fc5f5-jhzwl\" (UID: \"f68bf759-7bee-4078-95f2-38368fae9f0a\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-jhzwl"
Sep 09 21:14:54.624183 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:14:54.624062    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bcq9\" (UniqueName: \"kubernetes.io/projected/f68bf759-7bee-4078-95f2-38368fae9f0a-kube-api-access-4bcq9\") pod \"isvc-canary-promote-predictor-65ff4fc5f5-jhzwl\" (UID: \"f68bf759-7bee-4078-95f2-38368fae9f0a\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-jhzwl"
Sep 09 21:14:54.624183 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:14:54.624102    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f68bf759-7bee-4078-95f2-38368fae9f0a-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-65ff4fc5f5-jhzwl\" (UID: \"f68bf759-7bee-4078-95f2-38368fae9f0a\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-jhzwl"
Sep 09 21:14:54.624183 ip-10-0-138-224 kubenswrapper[2579]: E0909 21:14:54.624154    2579 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-promote-predictor-serving-cert: secret "isvc-canary-promote-predictor-serving-cert" not found
Sep 09 21:14:54.624183 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:14:54.624175    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f68bf759-7bee-4078-95f2-38368fae9f0a-cabundle-cert\") pod \"isvc-canary-promote-predictor-65ff4fc5f5-jhzwl\" (UID: \"f68bf759-7bee-4078-95f2-38368fae9f0a\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-jhzwl"
Sep 09 21:14:54.624482 ip-10-0-138-224 kubenswrapper[2579]: E0909 21:14:54.624222    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f68bf759-7bee-4078-95f2-38368fae9f0a-proxy-tls podName:f68bf759-7bee-4078-95f2-38368fae9f0a nodeName:}" failed. No retries permitted until 2026-09-09 21:14:55.124204949 +0000 UTC m=+1352.364299061 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/f68bf759-7bee-4078-95f2-38368fae9f0a-proxy-tls") pod "isvc-canary-promote-predictor-65ff4fc5f5-jhzwl" (UID: "f68bf759-7bee-4078-95f2-38368fae9f0a") : secret "isvc-canary-promote-predictor-serving-cert" not found
Sep 09 21:14:54.624559 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:14:54.624475    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f68bf759-7bee-4078-95f2-38368fae9f0a-kserve-provision-location\") pod \"isvc-canary-promote-predictor-65ff4fc5f5-jhzwl\" (UID: \"f68bf759-7bee-4078-95f2-38368fae9f0a\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-jhzwl"
Sep 09 21:14:54.624783 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:14:54.624758    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f68bf759-7bee-4078-95f2-38368fae9f0a-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-65ff4fc5f5-jhzwl\" (UID: \"f68bf759-7bee-4078-95f2-38368fae9f0a\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-jhzwl"
Sep 09 21:14:54.624878 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:14:54.624763    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f68bf759-7bee-4078-95f2-38368fae9f0a-cabundle-cert\") pod \"isvc-canary-promote-predictor-65ff4fc5f5-jhzwl\" (UID: \"f68bf759-7bee-4078-95f2-38368fae9f0a\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-jhzwl"
Sep 09 21:14:54.632272 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:14:54.632251    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bcq9\" (UniqueName: \"kubernetes.io/projected/f68bf759-7bee-4078-95f2-38368fae9f0a-kube-api-access-4bcq9\") pod \"isvc-canary-promote-predictor-65ff4fc5f5-jhzwl\" (UID: \"f68bf759-7bee-4078-95f2-38368fae9f0a\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-jhzwl"
Sep 09 21:14:54.701668 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:14:54.701642    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-zf64p"
Sep 09 21:14:54.980805 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:14:54.980783    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-zf64p"]
Sep 09 21:14:54.982627 ip-10-0-138-224 kubenswrapper[2579]: W0909 21:14:54.982597    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34726418_cf49_4e02_a3ae_c682369e582a.slice/crio-28d746ffdd3b927c8bf17ba66847fb9819a0cb6712196b855eac9691dcddb7c7 WatchSource:0}: Error finding container 28d746ffdd3b927c8bf17ba66847fb9819a0cb6712196b855eac9691dcddb7c7: Status 404 returned error can't find the container with id 28d746ffdd3b927c8bf17ba66847fb9819a0cb6712196b855eac9691dcddb7c7
Sep 09 21:14:55.129823 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:14:55.129781    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f68bf759-7bee-4078-95f2-38368fae9f0a-proxy-tls\") pod \"isvc-canary-promote-predictor-65ff4fc5f5-jhzwl\" (UID: \"f68bf759-7bee-4078-95f2-38368fae9f0a\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-jhzwl"
Sep 09 21:14:55.132197 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:14:55.132175    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f68bf759-7bee-4078-95f2-38368fae9f0a-proxy-tls\") pod \"isvc-canary-promote-predictor-65ff4fc5f5-jhzwl\" (UID: \"f68bf759-7bee-4078-95f2-38368fae9f0a\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-jhzwl"
Sep 09 21:14:55.291684 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:14:55.291600    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="31dd0d17-13fe-4305-88eb-77c3636749cd" path="/var/lib/kubelet/pods/31dd0d17-13fe-4305-88eb-77c3636749cd/volumes"
Sep 09 21:14:55.292110 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:14:55.292097    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dbcd4317-6eac-4497-aa10-5a2867fb1197" path="/var/lib/kubelet/pods/dbcd4317-6eac-4497-aa10-5a2867fb1197/volumes"
Sep 09 21:14:55.370417 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:14:55.370386    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-jhzwl"
Sep 09 21:14:55.470471 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:14:55.470417    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-zf64p" event={"ID":"34726418-cf49-4e02-a3ae-c682369e582a","Type":"ContainerStarted","Data":"4ce9804835f0fd307dc0ef219150a013305a13c561b5bbb763125739cc9e2a4d"}
Sep 09 21:14:55.470663 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:14:55.470504    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-zf64p" event={"ID":"34726418-cf49-4e02-a3ae-c682369e582a","Type":"ContainerStarted","Data":"28d746ffdd3b927c8bf17ba66847fb9819a0cb6712196b855eac9691dcddb7c7"}
Sep 09 21:14:55.647426 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:14:55.647339    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-jhzwl"]
Sep 09 21:14:55.650512 ip-10-0-138-224 kubenswrapper[2579]: W0909 21:14:55.650482    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf68bf759_7bee_4078_95f2_38368fae9f0a.slice/crio-f6c5297647c73a25b009c673792b59ed0caa79db32be83f48447e22f683f7edd WatchSource:0}: Error finding container f6c5297647c73a25b009c673792b59ed0caa79db32be83f48447e22f683f7edd: Status 404 returned error can't find the container with id f6c5297647c73a25b009c673792b59ed0caa79db32be83f48447e22f683f7edd
Sep 09 21:14:56.478642 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:14:56.478602    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-jhzwl" event={"ID":"f68bf759-7bee-4078-95f2-38368fae9f0a","Type":"ContainerStarted","Data":"3d7da8a6a0296cac6dd7f2ccc7039ca0777ad6f195d8e7086949c4abf4a68bec"}
Sep 09 21:14:56.479119 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:14:56.478667    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-jhzwl" event={"ID":"f68bf759-7bee-4078-95f2-38368fae9f0a","Type":"ContainerStarted","Data":"f6c5297647c73a25b009c673792b59ed0caa79db32be83f48447e22f683f7edd"}
Sep 09 21:14:59.493253 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:14:59.493219    2579 generic.go:356] "Generic (PLEG): container finished" podID="f68bf759-7bee-4078-95f2-38368fae9f0a" containerID="3d7da8a6a0296cac6dd7f2ccc7039ca0777ad6f195d8e7086949c4abf4a68bec" exitCode=0
Sep 09 21:14:59.493619 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:14:59.493293    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-jhzwl" event={"ID":"f68bf759-7bee-4078-95f2-38368fae9f0a","Type":"ContainerDied","Data":"3d7da8a6a0296cac6dd7f2ccc7039ca0777ad6f195d8e7086949c4abf4a68bec"}
Sep 09 21:14:59.495012 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:14:59.494989    2579 generic.go:356] "Generic (PLEG): container finished" podID="34726418-cf49-4e02-a3ae-c682369e582a" containerID="4ce9804835f0fd307dc0ef219150a013305a13c561b5bbb763125739cc9e2a4d" exitCode=0
Sep 09 21:14:59.495116 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:14:59.495023    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-zf64p" event={"ID":"34726418-cf49-4e02-a3ae-c682369e582a","Type":"ContainerDied","Data":"4ce9804835f0fd307dc0ef219150a013305a13c561b5bbb763125739cc9e2a4d"}
Sep 09 21:15:00.501657 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:15:00.501618    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-zf64p" event={"ID":"34726418-cf49-4e02-a3ae-c682369e582a","Type":"ContainerStarted","Data":"8fbc14260095b69cac450e5203cd53c57e59c67f48fca838edc4243e135b28ab"}
Sep 09 21:15:00.502074 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:15:00.501687    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-zf64p" event={"ID":"34726418-cf49-4e02-a3ae-c682369e582a","Type":"ContainerStarted","Data":"bd8e2edbdd63dd0137fb6464f0c43bcc4656dc5378a81f3b681ffb14504b0644"}
Sep 09 21:15:00.502074 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:15:00.502041    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-zf64p"
Sep 09 21:15:00.502203 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:15:00.502181    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-zf64p"
Sep 09 21:15:00.503674 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:15:00.503646    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-zf64p" podUID="34726418-cf49-4e02-a3ae-c682369e582a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Sep 09 21:15:00.503960 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:15:00.503942    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-jhzwl" event={"ID":"f68bf759-7bee-4078-95f2-38368fae9f0a","Type":"ContainerStarted","Data":"a3d1914ce3674c421c8c41eea785f5a6d637fcd8f911b42f38cce6718e5faa35"}
Sep 09 21:15:00.504031 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:15:00.503994    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-jhzwl" event={"ID":"f68bf759-7bee-4078-95f2-38368fae9f0a","Type":"ContainerStarted","Data":"2c9aa6ab62f6375088bab50b1aeac1f91aca1506dd2aefb07d114722d4edc1f0"}
Sep 09 21:15:00.504186 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:15:00.504169    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-jhzwl"
Sep 09 21:15:00.524706 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:15:00.524661    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-zf64p" podStartSLOduration=2.780352156 podStartE2EDuration="6.524647777s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.744295621s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:14:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:15:00.523974837 +0000 UTC m=+1357.764068982" watchObservedRunningTime="2026-09-09 21:15:00.524647777 +0000 UTC m=+1357.764741894"
Sep 09 21:15:00.548364 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:15:00.548313    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-jhzwl" podStartSLOduration=2.849524682 podStartE2EDuration="6.548297987s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.698773305s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:14:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:15:00.546088606 +0000 UTC m=+1357.786182747" watchObservedRunningTime="2026-09-09 21:15:00.548297987 +0000 UTC m=+1357.788392104"
Sep 09 21:15:01.509039 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:15:01.508981    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-zf64p" podUID="34726418-cf49-4e02-a3ae-c682369e582a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Sep 09 21:15:01.509431 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:15:01.509177    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-jhzwl"
Sep 09 21:15:01.510328 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:15:01.510304    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-jhzwl" podUID="f68bf759-7bee-4078-95f2-38368fae9f0a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Sep 09 21:15:02.513925 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:15:02.513886    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-jhzwl" podUID="f68bf759-7bee-4078-95f2-38368fae9f0a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Sep 09 21:15:06.513500 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:15:06.513469    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-zf64p"
Sep 09 21:15:06.514008 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:15:06.513973    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-zf64p" podUID="34726418-cf49-4e02-a3ae-c682369e582a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Sep 09 21:15:07.518829 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:15:07.518801    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-jhzwl"
Sep 09 21:15:07.519304 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:15:07.519279    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-jhzwl" podUID="f68bf759-7bee-4078-95f2-38368fae9f0a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Sep 09 21:15:16.514880 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:15:16.514839    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-zf64p" podUID="34726418-cf49-4e02-a3ae-c682369e582a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Sep 09 21:15:17.520271 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:15:17.520228    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-jhzwl" podUID="f68bf759-7bee-4078-95f2-38368fae9f0a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Sep 09 21:15:26.513893 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:15:26.513845    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-zf64p" podUID="34726418-cf49-4e02-a3ae-c682369e582a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Sep 09 21:15:27.520101 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:15:27.520062    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-jhzwl" podUID="f68bf759-7bee-4078-95f2-38368fae9f0a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Sep 09 21:15:36.514070 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:15:36.514027    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-zf64p" podUID="34726418-cf49-4e02-a3ae-c682369e582a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Sep 09 21:15:37.519273 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:15:37.519234    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-jhzwl" podUID="f68bf759-7bee-4078-95f2-38368fae9f0a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Sep 09 21:15:46.514313 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:15:46.514274    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-zf64p" podUID="34726418-cf49-4e02-a3ae-c682369e582a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Sep 09 21:15:47.519195 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:15:47.519157    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-jhzwl" podUID="f68bf759-7bee-4078-95f2-38368fae9f0a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Sep 09 21:15:56.514642 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:15:56.514601    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-zf64p" podUID="34726418-cf49-4e02-a3ae-c682369e582a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Sep 09 21:15:57.519347 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:15:57.519309    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-jhzwl" podUID="f68bf759-7bee-4078-95f2-38368fae9f0a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Sep 09 21:16:06.513974 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:16:06.513930    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-zf64p" podUID="34726418-cf49-4e02-a3ae-c682369e582a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Sep 09 21:16:07.520087 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:16:07.520047    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-jhzwl" podUID="f68bf759-7bee-4078-95f2-38368fae9f0a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Sep 09 21:16:15.291140 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:16:15.291060    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-zf64p"
Sep 09 21:16:16.286634 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:16:16.286602    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-jhzwl"
Sep 09 21:16:24.439996 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:16:24.439964    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-jhzwl"]
Sep 09 21:16:24.440486 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:16:24.440242    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-jhzwl" podUID="f68bf759-7bee-4078-95f2-38368fae9f0a" containerName="kserve-container" containerID="cri-o://2c9aa6ab62f6375088bab50b1aeac1f91aca1506dd2aefb07d114722d4edc1f0" gracePeriod=30
Sep 09 21:16:24.440486 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:16:24.440252    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-jhzwl" podUID="f68bf759-7bee-4078-95f2-38368fae9f0a" containerName="kube-rbac-proxy" containerID="cri-o://a3d1914ce3674c421c8c41eea785f5a6d637fcd8f911b42f38cce6718e5faa35" gracePeriod=30
Sep 09 21:16:24.896625 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:16:24.896549    2579 generic.go:356] "Generic (PLEG): container finished" podID="f68bf759-7bee-4078-95f2-38368fae9f0a" containerID="a3d1914ce3674c421c8c41eea785f5a6d637fcd8f911b42f38cce6718e5faa35" exitCode=2
Sep 09 21:16:24.896761 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:16:24.896620    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-jhzwl" event={"ID":"f68bf759-7bee-4078-95f2-38368fae9f0a","Type":"ContainerDied","Data":"a3d1914ce3674c421c8c41eea785f5a6d637fcd8f911b42f38cce6718e5faa35"}
Sep 09 21:16:26.286523 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:16:26.286469    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-jhzwl" podUID="f68bf759-7bee-4078-95f2-38368fae9f0a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Sep 09 21:16:27.515027 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:16:27.514986    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-jhzwl" podUID="f68bf759-7bee-4078-95f2-38368fae9f0a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.45:8643/healthz\": dial tcp 10.134.0.45:8643: connect: connection refused"
Sep 09 21:16:29.187314 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:16:29.187293    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-jhzwl"
Sep 09 21:16:29.237124 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:16:29.237065    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f68bf759-7bee-4078-95f2-38368fae9f0a-cabundle-cert\") pod \"f68bf759-7bee-4078-95f2-38368fae9f0a\" (UID: \"f68bf759-7bee-4078-95f2-38368fae9f0a\") "
Sep 09 21:16:29.237124 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:16:29.237106    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f68bf759-7bee-4078-95f2-38368fae9f0a-kserve-provision-location\") pod \"f68bf759-7bee-4078-95f2-38368fae9f0a\" (UID: \"f68bf759-7bee-4078-95f2-38368fae9f0a\") "
Sep 09 21:16:29.237288 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:16:29.237133    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4bcq9\" (UniqueName: \"kubernetes.io/projected/f68bf759-7bee-4078-95f2-38368fae9f0a-kube-api-access-4bcq9\") pod \"f68bf759-7bee-4078-95f2-38368fae9f0a\" (UID: \"f68bf759-7bee-4078-95f2-38368fae9f0a\") "
Sep 09 21:16:29.237288 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:16:29.237160    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f68bf759-7bee-4078-95f2-38368fae9f0a-kube-rbac-proxy-sar-config\") pod \"f68bf759-7bee-4078-95f2-38368fae9f0a\" (UID: \"f68bf759-7bee-4078-95f2-38368fae9f0a\") "
Sep 09 21:16:29.237288 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:16:29.237178    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f68bf759-7bee-4078-95f2-38368fae9f0a-proxy-tls\") pod \"f68bf759-7bee-4078-95f2-38368fae9f0a\" (UID: \"f68bf759-7bee-4078-95f2-38368fae9f0a\") "
Sep 09 21:16:29.237460 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:16:29.237416    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f68bf759-7bee-4078-95f2-38368fae9f0a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "f68bf759-7bee-4078-95f2-38368fae9f0a" (UID: "f68bf759-7bee-4078-95f2-38368fae9f0a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:16:29.237516 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:16:29.237476    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f68bf759-7bee-4078-95f2-38368fae9f0a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f68bf759-7bee-4078-95f2-38368fae9f0a" (UID: "f68bf759-7bee-4078-95f2-38368fae9f0a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:16:29.237516 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:16:29.237506    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f68bf759-7bee-4078-95f2-38368fae9f0a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "f68bf759-7bee-4078-95f2-38368fae9f0a" (UID: "f68bf759-7bee-4078-95f2-38368fae9f0a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:16:29.239386 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:16:29.239366    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f68bf759-7bee-4078-95f2-38368fae9f0a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f68bf759-7bee-4078-95f2-38368fae9f0a" (UID: "f68bf759-7bee-4078-95f2-38368fae9f0a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:16:29.239455 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:16:29.239417    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f68bf759-7bee-4078-95f2-38368fae9f0a-kube-api-access-4bcq9" (OuterVolumeSpecName: "kube-api-access-4bcq9") pod "f68bf759-7bee-4078-95f2-38368fae9f0a" (UID: "f68bf759-7bee-4078-95f2-38368fae9f0a"). InnerVolumeSpecName "kube-api-access-4bcq9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:16:29.337964 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:16:29.337938    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4bcq9\" (UniqueName: \"kubernetes.io/projected/f68bf759-7bee-4078-95f2-38368fae9f0a-kube-api-access-4bcq9\") on node \"ip-10-0-138-224.ec2.internal\" DevicePath \"\""
Sep 09 21:16:29.338096 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:16:29.337966    2579 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f68bf759-7bee-4078-95f2-38368fae9f0a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-224.ec2.internal\" DevicePath \"\""
Sep 09 21:16:29.338096 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:16:29.337981    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f68bf759-7bee-4078-95f2-38368fae9f0a-proxy-tls\") on node \"ip-10-0-138-224.ec2.internal\" DevicePath \"\""
Sep 09 21:16:29.338096 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:16:29.337994    2579 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f68bf759-7bee-4078-95f2-38368fae9f0a-cabundle-cert\") on node \"ip-10-0-138-224.ec2.internal\" DevicePath \"\""
Sep 09 21:16:29.338096 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:16:29.338007    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f68bf759-7bee-4078-95f2-38368fae9f0a-kserve-provision-location\") on node \"ip-10-0-138-224.ec2.internal\" DevicePath \"\""
Sep 09 21:16:29.921581 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:16:29.921546    2579 generic.go:356] "Generic (PLEG): container finished" podID="f68bf759-7bee-4078-95f2-38368fae9f0a" containerID="2c9aa6ab62f6375088bab50b1aeac1f91aca1506dd2aefb07d114722d4edc1f0" exitCode=0
Sep 09 21:16:29.921746 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:16:29.921611    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-jhzwl" event={"ID":"f68bf759-7bee-4078-95f2-38368fae9f0a","Type":"ContainerDied","Data":"2c9aa6ab62f6375088bab50b1aeac1f91aca1506dd2aefb07d114722d4edc1f0"}
Sep 09 21:16:29.921746 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:16:29.921617    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-jhzwl"
Sep 09 21:16:29.921746 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:16:29.921659    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-jhzwl" event={"ID":"f68bf759-7bee-4078-95f2-38368fae9f0a","Type":"ContainerDied","Data":"f6c5297647c73a25b009c673792b59ed0caa79db32be83f48447e22f683f7edd"}
Sep 09 21:16:29.921746 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:16:29.921685    2579 scope.go:117] "RemoveContainer" containerID="a3d1914ce3674c421c8c41eea785f5a6d637fcd8f911b42f38cce6718e5faa35"
Sep 09 21:16:29.930100 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:16:29.930083    2579 scope.go:117] "RemoveContainer" containerID="2c9aa6ab62f6375088bab50b1aeac1f91aca1506dd2aefb07d114722d4edc1f0"
Sep 09 21:16:29.937904 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:16:29.937885    2579 scope.go:117] "RemoveContainer" containerID="3d7da8a6a0296cac6dd7f2ccc7039ca0777ad6f195d8e7086949c4abf4a68bec"
Sep 09 21:16:29.942873 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:16:29.942853    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-jhzwl"]
Sep 09 21:16:29.945606 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:16:29.945581    2579 scope.go:117] "RemoveContainer" containerID="a3d1914ce3674c421c8c41eea785f5a6d637fcd8f911b42f38cce6718e5faa35"
Sep 09 21:16:29.945883 ip-10-0-138-224 kubenswrapper[2579]: E0909 21:16:29.945862    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3d1914ce3674c421c8c41eea785f5a6d637fcd8f911b42f38cce6718e5faa35\": container with ID starting with a3d1914ce3674c421c8c41eea785f5a6d637fcd8f911b42f38cce6718e5faa35 not found: ID does not exist" containerID="a3d1914ce3674c421c8c41eea785f5a6d637fcd8f911b42f38cce6718e5faa35"
Sep 09 21:16:29.945951 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:16:29.945892    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3d1914ce3674c421c8c41eea785f5a6d637fcd8f911b42f38cce6718e5faa35"} err="failed to get container status \"a3d1914ce3674c421c8c41eea785f5a6d637fcd8f911b42f38cce6718e5faa35\": rpc error: code = NotFound desc = could not find container \"a3d1914ce3674c421c8c41eea785f5a6d637fcd8f911b42f38cce6718e5faa35\": container with ID starting with a3d1914ce3674c421c8c41eea785f5a6d637fcd8f911b42f38cce6718e5faa35 not found: ID does not exist"
Sep 09 21:16:29.945951 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:16:29.945909    2579 scope.go:117] "RemoveContainer" containerID="2c9aa6ab62f6375088bab50b1aeac1f91aca1506dd2aefb07d114722d4edc1f0"
Sep 09 21:16:29.946171 ip-10-0-138-224 kubenswrapper[2579]: E0909 21:16:29.946153    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c9aa6ab62f6375088bab50b1aeac1f91aca1506dd2aefb07d114722d4edc1f0\": container with ID starting with 2c9aa6ab62f6375088bab50b1aeac1f91aca1506dd2aefb07d114722d4edc1f0 not found: ID does not exist" containerID="2c9aa6ab62f6375088bab50b1aeac1f91aca1506dd2aefb07d114722d4edc1f0"
Sep 09 21:16:29.946240 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:16:29.946181    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c9aa6ab62f6375088bab50b1aeac1f91aca1506dd2aefb07d114722d4edc1f0"} err="failed to get container status \"2c9aa6ab62f6375088bab50b1aeac1f91aca1506dd2aefb07d114722d4edc1f0\": rpc error: code = NotFound desc = could not find container \"2c9aa6ab62f6375088bab50b1aeac1f91aca1506dd2aefb07d114722d4edc1f0\": container with ID starting with 2c9aa6ab62f6375088bab50b1aeac1f91aca1506dd2aefb07d114722d4edc1f0 not found: ID does not exist"
Sep 09 21:16:29.946240 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:16:29.946203    2579 scope.go:117] "RemoveContainer" containerID="3d7da8a6a0296cac6dd7f2ccc7039ca0777ad6f195d8e7086949c4abf4a68bec"
Sep 09 21:16:29.946478 ip-10-0-138-224 kubenswrapper[2579]: E0909 21:16:29.946462    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d7da8a6a0296cac6dd7f2ccc7039ca0777ad6f195d8e7086949c4abf4a68bec\": container with ID starting with 3d7da8a6a0296cac6dd7f2ccc7039ca0777ad6f195d8e7086949c4abf4a68bec not found: ID does not exist" containerID="3d7da8a6a0296cac6dd7f2ccc7039ca0777ad6f195d8e7086949c4abf4a68bec"
Sep 09 21:16:29.946522 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:16:29.946483    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d7da8a6a0296cac6dd7f2ccc7039ca0777ad6f195d8e7086949c4abf4a68bec"} err="failed to get container status \"3d7da8a6a0296cac6dd7f2ccc7039ca0777ad6f195d8e7086949c4abf4a68bec\": rpc error: code = NotFound desc = could not find container \"3d7da8a6a0296cac6dd7f2ccc7039ca0777ad6f195d8e7086949c4abf4a68bec\": container with ID starting with 3d7da8a6a0296cac6dd7f2ccc7039ca0777ad6f195d8e7086949c4abf4a68bec not found: ID does not exist"
Sep 09 21:16:29.951213 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:16:29.951193    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-jhzwl"]
Sep 09 21:16:31.291098 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:16:31.291065    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f68bf759-7bee-4078-95f2-38368fae9f0a" path="/var/lib/kubelet/pods/f68bf759-7bee-4078-95f2-38368fae9f0a/volumes"
Sep 09 21:17:54.697700 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:17:54.697620    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-zf64p"]
Sep 09 21:17:54.698201 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:17:54.697854    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-zf64p" podUID="34726418-cf49-4e02-a3ae-c682369e582a" containerName="kserve-container" containerID="cri-o://bd8e2edbdd63dd0137fb6464f0c43bcc4656dc5378a81f3b681ffb14504b0644" gracePeriod=30
Sep 09 21:17:54.698201 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:17:54.697906    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-zf64p" podUID="34726418-cf49-4e02-a3ae-c682369e582a" containerName="kube-rbac-proxy" containerID="cri-o://8fbc14260095b69cac450e5203cd53c57e59c67f48fca838edc4243e135b28ab" gracePeriod=30
Sep 09 21:17:55.286101 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:17:55.286053    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-zf64p" podUID="34726418-cf49-4e02-a3ae-c682369e582a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Sep 09 21:17:55.288145 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:17:55.288116    2579 generic.go:356] "Generic (PLEG): container finished" podID="34726418-cf49-4e02-a3ae-c682369e582a" containerID="8fbc14260095b69cac450e5203cd53c57e59c67f48fca838edc4243e135b28ab" exitCode=2
Sep 09 21:17:55.291117 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:17:55.291095    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-zf64p" event={"ID":"34726418-cf49-4e02-a3ae-c682369e582a","Type":"ContainerDied","Data":"8fbc14260095b69cac450e5203cd53c57e59c67f48fca838edc4243e135b28ab"}
Sep 09 21:17:56.510190 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:17:56.510148    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-zf64p" podUID="34726418-cf49-4e02-a3ae-c682369e582a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.44:8643/healthz\": dial tcp 10.134.0.44:8643: connect: connection refused"
Sep 09 21:17:59.153874 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:17:59.153851    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-zf64p"
Sep 09 21:17:59.269320 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:17:59.269246    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x47s2\" (UniqueName: \"kubernetes.io/projected/34726418-cf49-4e02-a3ae-c682369e582a-kube-api-access-x47s2\") pod \"34726418-cf49-4e02-a3ae-c682369e582a\" (UID: \"34726418-cf49-4e02-a3ae-c682369e582a\") "
Sep 09 21:17:59.269320 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:17:59.269286    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/34726418-cf49-4e02-a3ae-c682369e582a-kserve-provision-location\") pod \"34726418-cf49-4e02-a3ae-c682369e582a\" (UID: \"34726418-cf49-4e02-a3ae-c682369e582a\") "
Sep 09 21:17:59.269320 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:17:59.269308    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/34726418-cf49-4e02-a3ae-c682369e582a-kube-rbac-proxy-sar-config\") pod \"34726418-cf49-4e02-a3ae-c682369e582a\" (UID: \"34726418-cf49-4e02-a3ae-c682369e582a\") "
Sep 09 21:17:59.269625 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:17:59.269336    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/34726418-cf49-4e02-a3ae-c682369e582a-cabundle-cert\") pod \"34726418-cf49-4e02-a3ae-c682369e582a\" (UID: \"34726418-cf49-4e02-a3ae-c682369e582a\") "
Sep 09 21:17:59.269625 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:17:59.269503    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/34726418-cf49-4e02-a3ae-c682369e582a-proxy-tls\") pod \"34726418-cf49-4e02-a3ae-c682369e582a\" (UID: \"34726418-cf49-4e02-a3ae-c682369e582a\") "
Sep 09 21:17:59.269722 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:17:59.269641    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34726418-cf49-4e02-a3ae-c682369e582a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "34726418-cf49-4e02-a3ae-c682369e582a" (UID: "34726418-cf49-4e02-a3ae-c682369e582a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:17:59.269722 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:17:59.269664    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34726418-cf49-4e02-a3ae-c682369e582a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "34726418-cf49-4e02-a3ae-c682369e582a" (UID: "34726418-cf49-4e02-a3ae-c682369e582a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:17:59.269722 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:17:59.269687    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34726418-cf49-4e02-a3ae-c682369e582a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "34726418-cf49-4e02-a3ae-c682369e582a" (UID: "34726418-cf49-4e02-a3ae-c682369e582a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:17:59.269846 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:17:59.269830    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/34726418-cf49-4e02-a3ae-c682369e582a-kserve-provision-location\") on node \"ip-10-0-138-224.ec2.internal\" DevicePath \"\""
Sep 09 21:17:59.269885 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:17:59.269849    2579 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/34726418-cf49-4e02-a3ae-c682369e582a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-224.ec2.internal\" DevicePath \"\""
Sep 09 21:17:59.269885 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:17:59.269859    2579 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/34726418-cf49-4e02-a3ae-c682369e582a-cabundle-cert\") on node \"ip-10-0-138-224.ec2.internal\" DevicePath \"\""
Sep 09 21:17:59.271627 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:17:59.271604    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34726418-cf49-4e02-a3ae-c682369e582a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "34726418-cf49-4e02-a3ae-c682369e582a" (UID: "34726418-cf49-4e02-a3ae-c682369e582a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:17:59.271726 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:17:59.271692    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34726418-cf49-4e02-a3ae-c682369e582a-kube-api-access-x47s2" (OuterVolumeSpecName: "kube-api-access-x47s2") pod "34726418-cf49-4e02-a3ae-c682369e582a" (UID: "34726418-cf49-4e02-a3ae-c682369e582a"). InnerVolumeSpecName "kube-api-access-x47s2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:17:59.306597 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:17:59.306568    2579 generic.go:356] "Generic (PLEG): container finished" podID="34726418-cf49-4e02-a3ae-c682369e582a" containerID="bd8e2edbdd63dd0137fb6464f0c43bcc4656dc5378a81f3b681ffb14504b0644" exitCode=0
Sep 09 21:17:59.306703 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:17:59.306625    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-zf64p"
Sep 09 21:17:59.306703 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:17:59.306649    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-zf64p" event={"ID":"34726418-cf49-4e02-a3ae-c682369e582a","Type":"ContainerDied","Data":"bd8e2edbdd63dd0137fb6464f0c43bcc4656dc5378a81f3b681ffb14504b0644"}
Sep 09 21:17:59.306829 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:17:59.306724    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-zf64p" event={"ID":"34726418-cf49-4e02-a3ae-c682369e582a","Type":"ContainerDied","Data":"28d746ffdd3b927c8bf17ba66847fb9819a0cb6712196b855eac9691dcddb7c7"}
Sep 09 21:17:59.306829 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:17:59.306766    2579 scope.go:117] "RemoveContainer" containerID="8fbc14260095b69cac450e5203cd53c57e59c67f48fca838edc4243e135b28ab"
Sep 09 21:17:59.315292 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:17:59.315271    2579 scope.go:117] "RemoveContainer" containerID="bd8e2edbdd63dd0137fb6464f0c43bcc4656dc5378a81f3b681ffb14504b0644"
Sep 09 21:17:59.322604 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:17:59.322587    2579 scope.go:117] "RemoveContainer" containerID="4ce9804835f0fd307dc0ef219150a013305a13c561b5bbb763125739cc9e2a4d"
Sep 09 21:17:59.327810 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:17:59.327790    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-zf64p"]
Sep 09 21:17:59.331207 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:17:59.331191    2579 scope.go:117] "RemoveContainer" containerID="8fbc14260095b69cac450e5203cd53c57e59c67f48fca838edc4243e135b28ab"
Sep 09 21:17:59.331549 ip-10-0-138-224 kubenswrapper[2579]: E0909 21:17:59.331525    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8fbc14260095b69cac450e5203cd53c57e59c67f48fca838edc4243e135b28ab\": container with ID starting with 8fbc14260095b69cac450e5203cd53c57e59c67f48fca838edc4243e135b28ab not found: ID does not exist" containerID="8fbc14260095b69cac450e5203cd53c57e59c67f48fca838edc4243e135b28ab"
Sep 09 21:17:59.331627 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:17:59.331560    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8fbc14260095b69cac450e5203cd53c57e59c67f48fca838edc4243e135b28ab"} err="failed to get container status \"8fbc14260095b69cac450e5203cd53c57e59c67f48fca838edc4243e135b28ab\": rpc error: code = NotFound desc = could not find container \"8fbc14260095b69cac450e5203cd53c57e59c67f48fca838edc4243e135b28ab\": container with ID starting with 8fbc14260095b69cac450e5203cd53c57e59c67f48fca838edc4243e135b28ab not found: ID does not exist"
Sep 09 21:17:59.331627 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:17:59.331579    2579 scope.go:117] "RemoveContainer" containerID="bd8e2edbdd63dd0137fb6464f0c43bcc4656dc5378a81f3b681ffb14504b0644"
Sep 09 21:17:59.331882 ip-10-0-138-224 kubenswrapper[2579]: E0909 21:17:59.331859    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd8e2edbdd63dd0137fb6464f0c43bcc4656dc5378a81f3b681ffb14504b0644\": container with ID starting with bd8e2edbdd63dd0137fb6464f0c43bcc4656dc5378a81f3b681ffb14504b0644 not found: ID does not exist" containerID="bd8e2edbdd63dd0137fb6464f0c43bcc4656dc5378a81f3b681ffb14504b0644"
Sep 09 21:17:59.331928 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:17:59.331891    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd8e2edbdd63dd0137fb6464f0c43bcc4656dc5378a81f3b681ffb14504b0644"} err="failed to get container status \"bd8e2edbdd63dd0137fb6464f0c43bcc4656dc5378a81f3b681ffb14504b0644\": rpc error: code = NotFound desc = could not find container \"bd8e2edbdd63dd0137fb6464f0c43bcc4656dc5378a81f3b681ffb14504b0644\": container with ID starting with bd8e2edbdd63dd0137fb6464f0c43bcc4656dc5378a81f3b681ffb14504b0644 not found: ID does not exist"
Sep 09 21:17:59.331967 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:17:59.331912    2579 scope.go:117] "RemoveContainer" containerID="4ce9804835f0fd307dc0ef219150a013305a13c561b5bbb763125739cc9e2a4d"
Sep 09 21:17:59.332211 ip-10-0-138-224 kubenswrapper[2579]: E0909 21:17:59.332194    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ce9804835f0fd307dc0ef219150a013305a13c561b5bbb763125739cc9e2a4d\": container with ID starting with 4ce9804835f0fd307dc0ef219150a013305a13c561b5bbb763125739cc9e2a4d not found: ID does not exist" containerID="4ce9804835f0fd307dc0ef219150a013305a13c561b5bbb763125739cc9e2a4d"
Sep 09 21:17:59.332262 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:17:59.332216    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ce9804835f0fd307dc0ef219150a013305a13c561b5bbb763125739cc9e2a4d"} err="failed to get container status \"4ce9804835f0fd307dc0ef219150a013305a13c561b5bbb763125739cc9e2a4d\": rpc error: code = NotFound desc = could not find container \"4ce9804835f0fd307dc0ef219150a013305a13c561b5bbb763125739cc9e2a4d\": container with ID starting with 4ce9804835f0fd307dc0ef219150a013305a13c561b5bbb763125739cc9e2a4d not found: ID does not exist"
Sep 09 21:17:59.332754 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:17:59.332738    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-zf64p"]
Sep 09 21:17:59.370685 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:17:59.370665    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x47s2\" (UniqueName: \"kubernetes.io/projected/34726418-cf49-4e02-a3ae-c682369e582a-kube-api-access-x47s2\") on node \"ip-10-0-138-224.ec2.internal\" DevicePath \"\""
Sep 09 21:17:59.370685 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:17:59.370684    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/34726418-cf49-4e02-a3ae-c682369e582a-proxy-tls\") on node \"ip-10-0-138-224.ec2.internal\" DevicePath \"\""
Sep 09 21:18:01.290729 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:18:01.290698    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="34726418-cf49-4e02-a3ae-c682369e582a" path="/var/lib/kubelet/pods/34726418-cf49-4e02-a3ae-c682369e582a/volumes"
Sep 09 21:18:19.277623 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:18:19.277588    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-577b6fff5f-7qr8c"]
Sep 09 21:18:19.279992 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:18:19.277919    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34726418-cf49-4e02-a3ae-c682369e582a" containerName="kserve-container"
Sep 09 21:18:19.279992 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:18:19.277930    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="34726418-cf49-4e02-a3ae-c682369e582a" containerName="kserve-container"
Sep 09 21:18:19.279992 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:18:19.277943    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f68bf759-7bee-4078-95f2-38368fae9f0a" containerName="kserve-container"
Sep 09 21:18:19.279992 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:18:19.277949    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="f68bf759-7bee-4078-95f2-38368fae9f0a" containerName="kserve-container"
Sep 09 21:18:19.279992 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:18:19.277959    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34726418-cf49-4e02-a3ae-c682369e582a" containerName="storage-initializer"
Sep 09 21:18:19.279992 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:18:19.277966    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="34726418-cf49-4e02-a3ae-c682369e582a" containerName="storage-initializer"
Sep 09 21:18:19.279992 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:18:19.277975    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f68bf759-7bee-4078-95f2-38368fae9f0a" containerName="kube-rbac-proxy"
Sep 09 21:18:19.279992 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:18:19.277980    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="f68bf759-7bee-4078-95f2-38368fae9f0a" containerName="kube-rbac-proxy"
Sep 09 21:18:19.279992 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:18:19.277988    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f68bf759-7bee-4078-95f2-38368fae9f0a" containerName="storage-initializer"
Sep 09 21:18:19.279992 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:18:19.277993    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="f68bf759-7bee-4078-95f2-38368fae9f0a" containerName="storage-initializer"
Sep 09 21:18:19.279992 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:18:19.278002    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34726418-cf49-4e02-a3ae-c682369e582a" containerName="kube-rbac-proxy"
Sep 09 21:18:19.279992 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:18:19.278007    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="34726418-cf49-4e02-a3ae-c682369e582a" containerName="kube-rbac-proxy"
Sep 09 21:18:19.279992 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:18:19.278060    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="34726418-cf49-4e02-a3ae-c682369e582a" containerName="kserve-container"
Sep 09 21:18:19.279992 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:18:19.278069    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="34726418-cf49-4e02-a3ae-c682369e582a" containerName="kube-rbac-proxy"
Sep 09 21:18:19.279992 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:18:19.278076    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="f68bf759-7bee-4078-95f2-38368fae9f0a" containerName="kube-rbac-proxy"
Sep 09 21:18:19.279992 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:18:19.278083    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="f68bf759-7bee-4078-95f2-38368fae9f0a" containerName="kserve-container"
Sep 09 21:18:19.281133 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:18:19.281117    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-577b6fff5f-7qr8c"
Sep 09 21:18:19.289387 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:18:19.289355    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-raw-cipn-predictor-serving-cert\""
Sep 09 21:18:19.289543 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:18:19.289471    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-tlmnz\""
Sep 09 21:18:19.289850 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:18:19.289832    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 09 21:18:19.290509 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:18:19.290486    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 09 21:18:19.290822 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:18:19.290802    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 09 21:18:19.290928 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:18:19.290825    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 09 21:18:19.291474 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:18:19.291434    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-raw-cipn-kube-rbac-proxy-sar-config\""
Sep 09 21:18:19.309986 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:18:19.309960    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-577b6fff5f-7qr8c"]
Sep 09 21:18:19.332982 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:18:19.332948    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5dfe596c-b72b-4d81-a229-c0c83adb7e8e-proxy-tls\") pod \"isvc-logger-raw-cipn-predictor-577b6fff5f-7qr8c\" (UID: \"5dfe596c-b72b-4d81-a229-c0c83adb7e8e\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-577b6fff5f-7qr8c"
Sep 09 21:18:19.332982 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:18:19.332984    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5dfe596c-b72b-4d81-a229-c0c83adb7e8e-kserve-provision-location\") pod \"isvc-logger-raw-cipn-predictor-577b6fff5f-7qr8c\" (UID: \"5dfe596c-b72b-4d81-a229-c0c83adb7e8e\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-577b6fff5f-7qr8c"
Sep 09 21:18:19.333178 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:18:19.333008    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5dfe596c-b72b-4d81-a229-c0c83adb7e8e-cabundle-cert\") pod \"isvc-logger-raw-cipn-predictor-577b6fff5f-7qr8c\" (UID: \"5dfe596c-b72b-4d81-a229-c0c83adb7e8e\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-577b6fff5f-7qr8c"
Sep 09 21:18:19.333178 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:18:19.333079    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5dfe596c-b72b-4d81-a229-c0c83adb7e8e-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-cipn-predictor-577b6fff5f-7qr8c\" (UID: \"5dfe596c-b72b-4d81-a229-c0c83adb7e8e\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-577b6fff5f-7qr8c"
Sep 09 21:18:19.333178 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:18:19.333101    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znw55\" (UniqueName: \"kubernetes.io/projected/5dfe596c-b72b-4d81-a229-c0c83adb7e8e-kube-api-access-znw55\") pod \"isvc-logger-raw-cipn-predictor-577b6fff5f-7qr8c\" (UID: \"5dfe596c-b72b-4d81-a229-c0c83adb7e8e\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-577b6fff5f-7qr8c"
Sep 09 21:18:19.434003 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:18:19.433953    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5dfe596c-b72b-4d81-a229-c0c83adb7e8e-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-cipn-predictor-577b6fff5f-7qr8c\" (UID: \"5dfe596c-b72b-4d81-a229-c0c83adb7e8e\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-577b6fff5f-7qr8c"
Sep 09 21:18:19.434003 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:18:19.434002    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znw55\" (UniqueName: \"kubernetes.io/projected/5dfe596c-b72b-4d81-a229-c0c83adb7e8e-kube-api-access-znw55\") pod \"isvc-logger-raw-cipn-predictor-577b6fff5f-7qr8c\" (UID: \"5dfe596c-b72b-4d81-a229-c0c83adb7e8e\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-577b6fff5f-7qr8c"
Sep 09 21:18:19.434264 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:18:19.434055    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5dfe596c-b72b-4d81-a229-c0c83adb7e8e-proxy-tls\") pod \"isvc-logger-raw-cipn-predictor-577b6fff5f-7qr8c\" (UID: \"5dfe596c-b72b-4d81-a229-c0c83adb7e8e\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-577b6fff5f-7qr8c"
Sep 09 21:18:19.434264 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:18:19.434077    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5dfe596c-b72b-4d81-a229-c0c83adb7e8e-kserve-provision-location\") pod \"isvc-logger-raw-cipn-predictor-577b6fff5f-7qr8c\" (UID: \"5dfe596c-b72b-4d81-a229-c0c83adb7e8e\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-577b6fff5f-7qr8c"
Sep 09 21:18:19.434264 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:18:19.434099    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5dfe596c-b72b-4d81-a229-c0c83adb7e8e-cabundle-cert\") pod \"isvc-logger-raw-cipn-predictor-577b6fff5f-7qr8c\" (UID: \"5dfe596c-b72b-4d81-a229-c0c83adb7e8e\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-577b6fff5f-7qr8c"
Sep 09 21:18:19.434264 ip-10-0-138-224 kubenswrapper[2579]: E0909 21:18:19.434209    2579 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-serving-cert: secret "isvc-logger-raw-cipn-predictor-serving-cert" not found
Sep 09 21:18:19.434420 ip-10-0-138-224 kubenswrapper[2579]: E0909 21:18:19.434274    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5dfe596c-b72b-4d81-a229-c0c83adb7e8e-proxy-tls podName:5dfe596c-b72b-4d81-a229-c0c83adb7e8e nodeName:}" failed. No retries permitted until 2026-09-09 21:18:19.934251977 +0000 UTC m=+1557.174346072 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/5dfe596c-b72b-4d81-a229-c0c83adb7e8e-proxy-tls") pod "isvc-logger-raw-cipn-predictor-577b6fff5f-7qr8c" (UID: "5dfe596c-b72b-4d81-a229-c0c83adb7e8e") : secret "isvc-logger-raw-cipn-predictor-serving-cert" not found
Sep 09 21:18:19.434556 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:18:19.434537    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5dfe596c-b72b-4d81-a229-c0c83adb7e8e-kserve-provision-location\") pod \"isvc-logger-raw-cipn-predictor-577b6fff5f-7qr8c\" (UID: \"5dfe596c-b72b-4d81-a229-c0c83adb7e8e\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-577b6fff5f-7qr8c"
Sep 09 21:18:19.434733 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:18:19.434714    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5dfe596c-b72b-4d81-a229-c0c83adb7e8e-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-cipn-predictor-577b6fff5f-7qr8c\" (UID: \"5dfe596c-b72b-4d81-a229-c0c83adb7e8e\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-577b6fff5f-7qr8c"
Sep 09 21:18:19.434770 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:18:19.434749    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5dfe596c-b72b-4d81-a229-c0c83adb7e8e-cabundle-cert\") pod \"isvc-logger-raw-cipn-predictor-577b6fff5f-7qr8c\" (UID: \"5dfe596c-b72b-4d81-a229-c0c83adb7e8e\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-577b6fff5f-7qr8c"
Sep 09 21:18:19.444423 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:18:19.444394    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-znw55\" (UniqueName: \"kubernetes.io/projected/5dfe596c-b72b-4d81-a229-c0c83adb7e8e-kube-api-access-znw55\") pod \"isvc-logger-raw-cipn-predictor-577b6fff5f-7qr8c\" (UID: \"5dfe596c-b72b-4d81-a229-c0c83adb7e8e\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-577b6fff5f-7qr8c"
Sep 09 21:18:19.937749 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:18:19.937706    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5dfe596c-b72b-4d81-a229-c0c83adb7e8e-proxy-tls\") pod \"isvc-logger-raw-cipn-predictor-577b6fff5f-7qr8c\" (UID: \"5dfe596c-b72b-4d81-a229-c0c83adb7e8e\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-577b6fff5f-7qr8c"
Sep 09 21:18:19.940258 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:18:19.940237    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5dfe596c-b72b-4d81-a229-c0c83adb7e8e-proxy-tls\") pod \"isvc-logger-raw-cipn-predictor-577b6fff5f-7qr8c\" (UID: \"5dfe596c-b72b-4d81-a229-c0c83adb7e8e\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-577b6fff5f-7qr8c"
Sep 09 21:18:20.191963 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:18:20.191885    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-577b6fff5f-7qr8c"
Sep 09 21:18:20.473507 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:18:20.473479    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-577b6fff5f-7qr8c"]
Sep 09 21:18:20.475301 ip-10-0-138-224 kubenswrapper[2579]: W0909 21:18:20.475274    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5dfe596c_b72b_4d81_a229_c0c83adb7e8e.slice/crio-1372f5b6dc166cee8634c67bc516ef2d108c5e5382bf2f24e687997da6d816f0 WatchSource:0}: Error finding container 1372f5b6dc166cee8634c67bc516ef2d108c5e5382bf2f24e687997da6d816f0: Status 404 returned error can't find the container with id 1372f5b6dc166cee8634c67bc516ef2d108c5e5382bf2f24e687997da6d816f0
Sep 09 21:18:21.403662 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:18:21.403618    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-577b6fff5f-7qr8c" event={"ID":"5dfe596c-b72b-4d81-a229-c0c83adb7e8e","Type":"ContainerStarted","Data":"dd6ecac7e68cc8e3ac8dfcb9565ca94ba75580adbe1e6b2549d206bc969050d2"}
Sep 09 21:18:21.403880 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:18:21.403693    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-577b6fff5f-7qr8c" event={"ID":"5dfe596c-b72b-4d81-a229-c0c83adb7e8e","Type":"ContainerStarted","Data":"1372f5b6dc166cee8634c67bc516ef2d108c5e5382bf2f24e687997da6d816f0"}
Sep 09 21:18:25.421757 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:18:25.421724    2579 generic.go:356] "Generic (PLEG): container finished" podID="5dfe596c-b72b-4d81-a229-c0c83adb7e8e" containerID="dd6ecac7e68cc8e3ac8dfcb9565ca94ba75580adbe1e6b2549d206bc969050d2" exitCode=0
Sep 09 21:18:25.422177 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:18:25.421767    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-577b6fff5f-7qr8c" event={"ID":"5dfe596c-b72b-4d81-a229-c0c83adb7e8e","Type":"ContainerDied","Data":"dd6ecac7e68cc8e3ac8dfcb9565ca94ba75580adbe1e6b2549d206bc969050d2"}
Sep 09 21:18:26.428423 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:18:26.428384    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-577b6fff5f-7qr8c" event={"ID":"5dfe596c-b72b-4d81-a229-c0c83adb7e8e","Type":"ContainerStarted","Data":"cdf7c0bc274ca398e2573dd5349448d41cbcb6e3a907e3529b89ee030921ffd9"}
Sep 09 21:18:26.428910 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:18:26.428504    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-577b6fff5f-7qr8c" event={"ID":"5dfe596c-b72b-4d81-a229-c0c83adb7e8e","Type":"ContainerStarted","Data":"f489cbf0f6a90f4b110caae128c2cbde865eb740882fc64eb46afe671a20771a"}
Sep 09 21:18:26.428910 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:18:26.428530    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-577b6fff5f-7qr8c" event={"ID":"5dfe596c-b72b-4d81-a229-c0c83adb7e8e","Type":"ContainerStarted","Data":"ea89ccd332f4653606a7438cfb6f9c8bdd3c003e7b72729207b23030a218c3b1"}
Sep 09 21:18:26.428910 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:18:26.428871    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-577b6fff5f-7qr8c"
Sep 09 21:18:26.429089 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:18:26.428921    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-577b6fff5f-7qr8c"
Sep 09 21:18:26.430221 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:18:26.430192    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-577b6fff5f-7qr8c" podUID="5dfe596c-b72b-4d81-a229-c0c83adb7e8e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Sep 09 21:18:26.453658 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:18:26.453603    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-577b6fff5f-7qr8c" podStartSLOduration=3.475988388 podStartE2EDuration="7.45358872s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.977600332s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:18:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:18:26.451877437 +0000 UTC m=+1563.691971556" watchObservedRunningTime="2026-09-09 21:18:26.45358872 +0000 UTC m=+1563.693682837"
Sep 09 21:18:27.433609 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:18:27.433564    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-577b6fff5f-7qr8c"
Sep 09 21:18:27.434077 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:18:27.433751    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-577b6fff5f-7qr8c" podUID="5dfe596c-b72b-4d81-a229-c0c83adb7e8e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Sep 09 21:18:27.434818 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:18:27.434795    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-577b6fff5f-7qr8c" podUID="5dfe596c-b72b-4d81-a229-c0c83adb7e8e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 21:18:28.438584 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:18:28.438536    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-577b6fff5f-7qr8c" podUID="5dfe596c-b72b-4d81-a229-c0c83adb7e8e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Sep 09 21:18:28.438992 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:18:28.438922    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-577b6fff5f-7qr8c" podUID="5dfe596c-b72b-4d81-a229-c0c83adb7e8e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 21:18:33.442428 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:18:33.442391    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-577b6fff5f-7qr8c"
Sep 09 21:18:33.443029 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:18:33.443000    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-577b6fff5f-7qr8c" podUID="5dfe596c-b72b-4d81-a229-c0c83adb7e8e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Sep 09 21:18:33.443330 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:18:33.443298    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-577b6fff5f-7qr8c" podUID="5dfe596c-b72b-4d81-a229-c0c83adb7e8e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 21:18:43.443192 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:18:43.443144    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-577b6fff5f-7qr8c" podUID="5dfe596c-b72b-4d81-a229-c0c83adb7e8e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Sep 09 21:18:43.443678 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:18:43.443656    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-577b6fff5f-7qr8c" podUID="5dfe596c-b72b-4d81-a229-c0c83adb7e8e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 21:18:53.443678 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:18:53.443632    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-577b6fff5f-7qr8c" podUID="5dfe596c-b72b-4d81-a229-c0c83adb7e8e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Sep 09 21:18:53.444162 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:18:53.444141    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-577b6fff5f-7qr8c" podUID="5dfe596c-b72b-4d81-a229-c0c83adb7e8e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 21:19:03.443311 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:19:03.443268    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-577b6fff5f-7qr8c" podUID="5dfe596c-b72b-4d81-a229-c0c83adb7e8e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Sep 09 21:19:03.443884 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:19:03.443816    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-577b6fff5f-7qr8c" podUID="5dfe596c-b72b-4d81-a229-c0c83adb7e8e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 21:19:13.443490 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:19:13.443425    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-577b6fff5f-7qr8c" podUID="5dfe596c-b72b-4d81-a229-c0c83adb7e8e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Sep 09 21:19:13.444008 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:19:13.443984    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-577b6fff5f-7qr8c" podUID="5dfe596c-b72b-4d81-a229-c0c83adb7e8e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 21:19:23.443680 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:19:23.443578    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-577b6fff5f-7qr8c" podUID="5dfe596c-b72b-4d81-a229-c0c83adb7e8e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Sep 09 21:19:23.444138 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:19:23.444114    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-577b6fff5f-7qr8c" podUID="5dfe596c-b72b-4d81-a229-c0c83adb7e8e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 21:19:33.443073 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:19:33.443035    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-577b6fff5f-7qr8c" podUID="5dfe596c-b72b-4d81-a229-c0c83adb7e8e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Sep 09 21:19:33.443557 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:19:33.443352    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-577b6fff5f-7qr8c" podUID="5dfe596c-b72b-4d81-a229-c0c83adb7e8e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 21:19:38.285831 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:19:38.285784    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-577b6fff5f-7qr8c" podUID="5dfe596c-b72b-4d81-a229-c0c83adb7e8e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Sep 09 21:19:38.286409 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:19:38.286383    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-577b6fff5f-7qr8c" podUID="5dfe596c-b72b-4d81-a229-c0c83adb7e8e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 21:19:48.286644 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:19:48.286613    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-577b6fff5f-7qr8c"
Sep 09 21:19:48.287027 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:19:48.286712    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-577b6fff5f-7qr8c"
Sep 09 21:19:54.500998 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:19:54.500961    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-577b6fff5f-7qr8c"]
Sep 09 21:19:54.501403 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:19:54.501314    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-577b6fff5f-7qr8c" podUID="5dfe596c-b72b-4d81-a229-c0c83adb7e8e" containerName="kserve-container" containerID="cri-o://ea89ccd332f4653606a7438cfb6f9c8bdd3c003e7b72729207b23030a218c3b1" gracePeriod=30
Sep 09 21:19:54.501403 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:19:54.501372    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-577b6fff5f-7qr8c" podUID="5dfe596c-b72b-4d81-a229-c0c83adb7e8e" containerName="agent" containerID="cri-o://cdf7c0bc274ca398e2573dd5349448d41cbcb6e3a907e3529b89ee030921ffd9" gracePeriod=30
Sep 09 21:19:54.501560 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:19:54.501408    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-577b6fff5f-7qr8c" podUID="5dfe596c-b72b-4d81-a229-c0c83adb7e8e" containerName="kube-rbac-proxy" containerID="cri-o://f489cbf0f6a90f4b110caae128c2cbde865eb740882fc64eb46afe671a20771a" gracePeriod=30
Sep 09 21:19:54.832183 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:19:54.832093    2579 generic.go:356] "Generic (PLEG): container finished" podID="5dfe596c-b72b-4d81-a229-c0c83adb7e8e" containerID="f489cbf0f6a90f4b110caae128c2cbde865eb740882fc64eb46afe671a20771a" exitCode=2
Sep 09 21:19:54.832183 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:19:54.832144    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-577b6fff5f-7qr8c" event={"ID":"5dfe596c-b72b-4d81-a229-c0c83adb7e8e","Type":"ContainerDied","Data":"f489cbf0f6a90f4b110caae128c2cbde865eb740882fc64eb46afe671a20771a"}
Sep 09 21:19:58.286587 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:19:58.286530    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-577b6fff5f-7qr8c" podUID="5dfe596c-b72b-4d81-a229-c0c83adb7e8e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Sep 09 21:19:58.286999 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:19:58.286841    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-577b6fff5f-7qr8c" podUID="5dfe596c-b72b-4d81-a229-c0c83adb7e8e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 21:19:58.439177 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:19:58.439130    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-577b6fff5f-7qr8c" podUID="5dfe596c-b72b-4d81-a229-c0c83adb7e8e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.46:8643/healthz\": dial tcp 10.134.0.46:8643: connect: connection refused"
Sep 09 21:19:59.858020 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:19:59.857984    2579 generic.go:356] "Generic (PLEG): container finished" podID="5dfe596c-b72b-4d81-a229-c0c83adb7e8e" containerID="ea89ccd332f4653606a7438cfb6f9c8bdd3c003e7b72729207b23030a218c3b1" exitCode=0
Sep 09 21:19:59.858020 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:19:59.858024    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-577b6fff5f-7qr8c" event={"ID":"5dfe596c-b72b-4d81-a229-c0c83adb7e8e","Type":"ContainerDied","Data":"ea89ccd332f4653606a7438cfb6f9c8bdd3c003e7b72729207b23030a218c3b1"}
Sep 09 21:20:03.439087 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:20:03.439038    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-577b6fff5f-7qr8c" podUID="5dfe596c-b72b-4d81-a229-c0c83adb7e8e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.46:8643/healthz\": dial tcp 10.134.0.46:8643: connect: connection refused"
Sep 09 21:20:08.285971 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:20:08.285919    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-577b6fff5f-7qr8c" podUID="5dfe596c-b72b-4d81-a229-c0c83adb7e8e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Sep 09 21:20:08.286428 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:20:08.286246    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-577b6fff5f-7qr8c" podUID="5dfe596c-b72b-4d81-a229-c0c83adb7e8e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 21:20:08.439350 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:20:08.439299    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-577b6fff5f-7qr8c" podUID="5dfe596c-b72b-4d81-a229-c0c83adb7e8e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.46:8643/healthz\": dial tcp 10.134.0.46:8643: connect: connection refused"
Sep 09 21:20:08.439564 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:20:08.439465    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-577b6fff5f-7qr8c"
Sep 09 21:20:11.984297 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:20:11.984266    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-577b6fff5f-7qr8c_5dfe596c-b72b-4d81-a229-c0c83adb7e8e/kserve-container/0.log"
Sep 09 21:20:12.029478 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:20:12.029429    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-577b6fff5f-7qr8c_5dfe596c-b72b-4d81-a229-c0c83adb7e8e/kube-rbac-proxy/0.log"
Sep 09 21:20:12.045804 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:20:12.045779    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-577b6fff5f-7qr8c_5dfe596c-b72b-4d81-a229-c0c83adb7e8e/agent/0.log"
Sep 09 21:20:12.060998 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:20:12.060975    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-577b6fff5f-7qr8c_5dfe596c-b72b-4d81-a229-c0c83adb7e8e/storage-initializer/0.log"
Sep 09 21:20:12.970172 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:20:12.970142    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-577b6fff5f-7qr8c_5dfe596c-b72b-4d81-a229-c0c83adb7e8e/kserve-container/0.log"
Sep 09 21:20:12.993064 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:20:12.993035    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-577b6fff5f-7qr8c_5dfe596c-b72b-4d81-a229-c0c83adb7e8e/kube-rbac-proxy/0.log"
Sep 09 21:20:13.005987 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:20:13.005955    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-577b6fff5f-7qr8c_5dfe596c-b72b-4d81-a229-c0c83adb7e8e/agent/0.log"
Sep 09 21:20:13.020897 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:20:13.020855    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-577b6fff5f-7qr8c_5dfe596c-b72b-4d81-a229-c0c83adb7e8e/storage-initializer/0.log"
Sep 09 21:20:13.438832 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:20:13.438794    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-577b6fff5f-7qr8c" podUID="5dfe596c-b72b-4d81-a229-c0c83adb7e8e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.46:8643/healthz\": dial tcp 10.134.0.46:8643: connect: connection refused"
Sep 09 21:20:13.937028 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:20:13.936993    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-577b6fff5f-7qr8c_5dfe596c-b72b-4d81-a229-c0c83adb7e8e/kserve-container/0.log"
Sep 09 21:20:13.949500 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:20:13.949464    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-577b6fff5f-7qr8c_5dfe596c-b72b-4d81-a229-c0c83adb7e8e/kube-rbac-proxy/0.log"
Sep 09 21:20:13.962771 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:20:13.962746    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-577b6fff5f-7qr8c_5dfe596c-b72b-4d81-a229-c0c83adb7e8e/agent/0.log"
Sep 09 21:20:13.974171 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:20:13.974144    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-577b6fff5f-7qr8c_5dfe596c-b72b-4d81-a229-c0c83adb7e8e/storage-initializer/0.log"
Sep 09 21:20:14.827005 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:20:14.826971    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-577b6fff5f-7qr8c_5dfe596c-b72b-4d81-a229-c0c83adb7e8e/kserve-container/0.log"
Sep 09 21:20:14.840965 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:20:14.840935    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-577b6fff5f-7qr8c_5dfe596c-b72b-4d81-a229-c0c83adb7e8e/kube-rbac-proxy/0.log"
Sep 09 21:20:14.861599 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:20:14.861573    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-577b6fff5f-7qr8c_5dfe596c-b72b-4d81-a229-c0c83adb7e8e/agent/0.log"
Sep 09 21:20:14.877214 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:20:14.877191    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-577b6fff5f-7qr8c_5dfe596c-b72b-4d81-a229-c0c83adb7e8e/storage-initializer/0.log"
Sep 09 21:20:15.738036 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:20:15.738005    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-577b6fff5f-7qr8c_5dfe596c-b72b-4d81-a229-c0c83adb7e8e/kserve-container/0.log"
Sep 09 21:20:15.772733 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:20:15.772706    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-577b6fff5f-7qr8c_5dfe596c-b72b-4d81-a229-c0c83adb7e8e/kube-rbac-proxy/0.log"
Sep 09 21:20:15.792718 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:20:15.792693    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-577b6fff5f-7qr8c_5dfe596c-b72b-4d81-a229-c0c83adb7e8e/agent/0.log"
Sep 09 21:20:15.808483 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:20:15.808434    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-577b6fff5f-7qr8c_5dfe596c-b72b-4d81-a229-c0c83adb7e8e/storage-initializer/0.log"
Sep 09 21:20:16.647743 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:20:16.647715    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-577b6fff5f-7qr8c_5dfe596c-b72b-4d81-a229-c0c83adb7e8e/kserve-container/0.log"
Sep 09 21:20:16.664498 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:20:16.664467    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-577b6fff5f-7qr8c_5dfe596c-b72b-4d81-a229-c0c83adb7e8e/kube-rbac-proxy/0.log"
Sep 09 21:20:16.675328 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:20:16.675301    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-577b6fff5f-7qr8c_5dfe596c-b72b-4d81-a229-c0c83adb7e8e/agent/0.log"
Sep 09 21:20:16.685925 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:20:16.685900    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-577b6fff5f-7qr8c_5dfe596c-b72b-4d81-a229-c0c83adb7e8e/storage-initializer/0.log"
Sep 09 21:20:17.560034 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:20:17.560006    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-577b6fff5f-7qr8c_5dfe596c-b72b-4d81-a229-c0c83adb7e8e/kserve-container/0.log"
Sep 09 21:20:17.574157 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:20:17.574132    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-577b6fff5f-7qr8c_5dfe596c-b72b-4d81-a229-c0c83adb7e8e/kube-rbac-proxy/0.log"
Sep 09 21:20:17.584977 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:20:17.584951    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-577b6fff5f-7qr8c_5dfe596c-b72b-4d81-a229-c0c83adb7e8e/agent/0.log"
Sep 09 21:20:17.596369 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:20:17.596347    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-577b6fff5f-7qr8c_5dfe596c-b72b-4d81-a229-c0c83adb7e8e/storage-initializer/0.log"
Sep 09 21:20:18.286212 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:20:18.286166    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-577b6fff5f-7qr8c" podUID="5dfe596c-b72b-4d81-a229-c0c83adb7e8e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Sep 09 21:20:18.286707 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:20:18.286346    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-577b6fff5f-7qr8c"
Sep 09 21:20:18.286707 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:20:18.286578    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-577b6fff5f-7qr8c" podUID="5dfe596c-b72b-4d81-a229-c0c83adb7e8e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 21:20:18.286707 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:20:18.286660    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-577b6fff5f-7qr8c"
Sep 09 21:20:18.439716 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:20:18.439672    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-577b6fff5f-7qr8c" podUID="5dfe596c-b72b-4d81-a229-c0c83adb7e8e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.46:8643/healthz\": dial tcp 10.134.0.46:8643: connect: connection refused"
Sep 09 21:20:18.483145 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:20:18.483110    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-577b6fff5f-7qr8c_5dfe596c-b72b-4d81-a229-c0c83adb7e8e/kserve-container/0.log"
Sep 09 21:20:18.501389 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:20:18.501359    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-577b6fff5f-7qr8c_5dfe596c-b72b-4d81-a229-c0c83adb7e8e/kube-rbac-proxy/0.log"
Sep 09 21:20:18.528878 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:20:18.528824    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-577b6fff5f-7qr8c_5dfe596c-b72b-4d81-a229-c0c83adb7e8e/agent/0.log"
Sep 09 21:20:18.538194 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:20:18.538128    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-577b6fff5f-7qr8c_5dfe596c-b72b-4d81-a229-c0c83adb7e8e/storage-initializer/0.log"
Sep 09 21:20:19.390765 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:20:19.390735    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-577b6fff5f-7qr8c_5dfe596c-b72b-4d81-a229-c0c83adb7e8e/kserve-container/0.log"
Sep 09 21:20:19.409241 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:20:19.409217    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-577b6fff5f-7qr8c_5dfe596c-b72b-4d81-a229-c0c83adb7e8e/kube-rbac-proxy/0.log"
Sep 09 21:20:19.429535 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:20:19.429511    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-577b6fff5f-7qr8c_5dfe596c-b72b-4d81-a229-c0c83adb7e8e/agent/0.log"
Sep 09 21:20:19.450492 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:20:19.450466    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-577b6fff5f-7qr8c_5dfe596c-b72b-4d81-a229-c0c83adb7e8e/storage-initializer/0.log"
Sep 09 21:20:20.300743 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:20:20.300711    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-577b6fff5f-7qr8c_5dfe596c-b72b-4d81-a229-c0c83adb7e8e/kserve-container/0.log"
Sep 09 21:20:20.315384 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:20:20.315360    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-577b6fff5f-7qr8c_5dfe596c-b72b-4d81-a229-c0c83adb7e8e/kube-rbac-proxy/0.log"
Sep 09 21:20:20.329186 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:20:20.329163    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-577b6fff5f-7qr8c_5dfe596c-b72b-4d81-a229-c0c83adb7e8e/agent/0.log"
Sep 09 21:20:20.347567 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:20:20.347540    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-577b6fff5f-7qr8c_5dfe596c-b72b-4d81-a229-c0c83adb7e8e/storage-initializer/0.log"
Sep 09 21:20:21.227469 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:20:21.227418    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-577b6fff5f-7qr8c_5dfe596c-b72b-4d81-a229-c0c83adb7e8e/kserve-container/0.log"
Sep 09 21:20:21.242574 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:20:21.242550    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-577b6fff5f-7qr8c_5dfe596c-b72b-4d81-a229-c0c83adb7e8e/kube-rbac-proxy/0.log"
Sep 09 21:20:21.254449 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:20:21.254409    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-577b6fff5f-7qr8c_5dfe596c-b72b-4d81-a229-c0c83adb7e8e/agent/0.log"
Sep 09 21:20:21.266155 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:20:21.266126    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-577b6fff5f-7qr8c_5dfe596c-b72b-4d81-a229-c0c83adb7e8e/storage-initializer/0.log"
Sep 09 21:20:22.155341 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:20:22.155309    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-577b6fff5f-7qr8c_5dfe596c-b72b-4d81-a229-c0c83adb7e8e/kserve-container/0.log"
Sep 09 21:20:22.171780 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:20:22.171742    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-577b6fff5f-7qr8c_5dfe596c-b72b-4d81-a229-c0c83adb7e8e/kube-rbac-proxy/0.log"
Sep 09 21:20:22.188580 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:20:22.188559    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-577b6fff5f-7qr8c_5dfe596c-b72b-4d81-a229-c0c83adb7e8e/agent/0.log"
Sep 09 21:20:22.202015 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:20:22.201980    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-577b6fff5f-7qr8c_5dfe596c-b72b-4d81-a229-c0c83adb7e8e/storage-initializer/0.log"
Sep 09 21:20:23.146775 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:20:23.146746    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-577b6fff5f-7qr8c_5dfe596c-b72b-4d81-a229-c0c83adb7e8e/kserve-container/0.log"
Sep 09 21:20:23.162937 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:20:23.162911    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-577b6fff5f-7qr8c_5dfe596c-b72b-4d81-a229-c0c83adb7e8e/kube-rbac-proxy/0.log"
Sep 09 21:20:23.194500 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:20:23.194471    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-577b6fff5f-7qr8c_5dfe596c-b72b-4d81-a229-c0c83adb7e8e/agent/0.log"
Sep 09 21:20:23.217521 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:20:23.217496    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-577b6fff5f-7qr8c_5dfe596c-b72b-4d81-a229-c0c83adb7e8e/storage-initializer/0.log"
Sep 09 21:20:23.439114 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:20:23.439079    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-577b6fff5f-7qr8c" podUID="5dfe596c-b72b-4d81-a229-c0c83adb7e8e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.46:8643/healthz\": dial tcp 10.134.0.46:8643: connect: connection refused"
Sep 09 21:20:24.130225 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:20:24.130192    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-577b6fff5f-7qr8c_5dfe596c-b72b-4d81-a229-c0c83adb7e8e/kserve-container/0.log"
Sep 09 21:20:24.143423 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:20:24.143398    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-577b6fff5f-7qr8c_5dfe596c-b72b-4d81-a229-c0c83adb7e8e/kube-rbac-proxy/0.log"
Sep 09 21:20:24.154408 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:20:24.154390    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-577b6fff5f-7qr8c_5dfe596c-b72b-4d81-a229-c0c83adb7e8e/agent/0.log"
Sep 09 21:20:24.164978 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:20:24.164954    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-577b6fff5f-7qr8c_5dfe596c-b72b-4d81-a229-c0c83adb7e8e/storage-initializer/0.log"
Sep 09 21:20:24.651772 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:20:24.651736    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-577b6fff5f-7qr8c"
Sep 09 21:20:24.689635 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:20:24.689603    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5dfe596c-b72b-4d81-a229-c0c83adb7e8e-proxy-tls\") pod \"5dfe596c-b72b-4d81-a229-c0c83adb7e8e\" (UID: \"5dfe596c-b72b-4d81-a229-c0c83adb7e8e\") "
Sep 09 21:20:24.689793 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:20:24.689656    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5dfe596c-b72b-4d81-a229-c0c83adb7e8e-kube-rbac-proxy-sar-config\") pod \"5dfe596c-b72b-4d81-a229-c0c83adb7e8e\" (UID: \"5dfe596c-b72b-4d81-a229-c0c83adb7e8e\") "
Sep 09 21:20:24.689793 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:20:24.689766    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5dfe596c-b72b-4d81-a229-c0c83adb7e8e-kserve-provision-location\") pod \"5dfe596c-b72b-4d81-a229-c0c83adb7e8e\" (UID: \"5dfe596c-b72b-4d81-a229-c0c83adb7e8e\") "
Sep 09 21:20:24.689912 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:20:24.689794    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-znw55\" (UniqueName: \"kubernetes.io/projected/5dfe596c-b72b-4d81-a229-c0c83adb7e8e-kube-api-access-znw55\") pod \"5dfe596c-b72b-4d81-a229-c0c83adb7e8e\" (UID: \"5dfe596c-b72b-4d81-a229-c0c83adb7e8e\") "
Sep 09 21:20:24.689912 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:20:24.689846    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5dfe596c-b72b-4d81-a229-c0c83adb7e8e-cabundle-cert\") pod \"5dfe596c-b72b-4d81-a229-c0c83adb7e8e\" (UID: \"5dfe596c-b72b-4d81-a229-c0c83adb7e8e\") "
Sep 09 21:20:24.690068 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:20:24.690040    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5dfe596c-b72b-4d81-a229-c0c83adb7e8e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "5dfe596c-b72b-4d81-a229-c0c83adb7e8e" (UID: "5dfe596c-b72b-4d81-a229-c0c83adb7e8e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:20:24.690131 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:20:24.690082    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5dfe596c-b72b-4d81-a229-c0c83adb7e8e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "5dfe596c-b72b-4d81-a229-c0c83adb7e8e" (UID: "5dfe596c-b72b-4d81-a229-c0c83adb7e8e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:20:24.690561 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:20:24.690537    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5dfe596c-b72b-4d81-a229-c0c83adb7e8e-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "5dfe596c-b72b-4d81-a229-c0c83adb7e8e" (UID: "5dfe596c-b72b-4d81-a229-c0c83adb7e8e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:20:24.691944 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:20:24.691919    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5dfe596c-b72b-4d81-a229-c0c83adb7e8e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "5dfe596c-b72b-4d81-a229-c0c83adb7e8e" (UID: "5dfe596c-b72b-4d81-a229-c0c83adb7e8e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:20:24.692032 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:20:24.691956    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5dfe596c-b72b-4d81-a229-c0c83adb7e8e-kube-api-access-znw55" (OuterVolumeSpecName: "kube-api-access-znw55") pod "5dfe596c-b72b-4d81-a229-c0c83adb7e8e" (UID: "5dfe596c-b72b-4d81-a229-c0c83adb7e8e"). InnerVolumeSpecName "kube-api-access-znw55". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:20:24.790563 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:20:24.790483    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5dfe596c-b72b-4d81-a229-c0c83adb7e8e-proxy-tls\") on node \"ip-10-0-138-224.ec2.internal\" DevicePath \"\""
Sep 09 21:20:24.790563 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:20:24.790513    2579 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5dfe596c-b72b-4d81-a229-c0c83adb7e8e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-224.ec2.internal\" DevicePath \"\""
Sep 09 21:20:24.790563 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:20:24.790524    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5dfe596c-b72b-4d81-a229-c0c83adb7e8e-kserve-provision-location\") on node \"ip-10-0-138-224.ec2.internal\" DevicePath \"\""
Sep 09 21:20:24.790563 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:20:24.790534    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-znw55\" (UniqueName: \"kubernetes.io/projected/5dfe596c-b72b-4d81-a229-c0c83adb7e8e-kube-api-access-znw55\") on node \"ip-10-0-138-224.ec2.internal\" DevicePath \"\""
Sep 09 21:20:24.790563 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:20:24.790542    2579 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5dfe596c-b72b-4d81-a229-c0c83adb7e8e-cabundle-cert\") on node \"ip-10-0-138-224.ec2.internal\" DevicePath \"\""
Sep 09 21:20:24.975596 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:20:24.975557    2579 generic.go:356] "Generic (PLEG): container finished" podID="5dfe596c-b72b-4d81-a229-c0c83adb7e8e" containerID="cdf7c0bc274ca398e2573dd5349448d41cbcb6e3a907e3529b89ee030921ffd9" exitCode=0
Sep 09 21:20:24.975812 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:20:24.975616    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-577b6fff5f-7qr8c"
Sep 09 21:20:24.975812 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:20:24.975641    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-577b6fff5f-7qr8c" event={"ID":"5dfe596c-b72b-4d81-a229-c0c83adb7e8e","Type":"ContainerDied","Data":"cdf7c0bc274ca398e2573dd5349448d41cbcb6e3a907e3529b89ee030921ffd9"}
Sep 09 21:20:24.975812 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:20:24.975755    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-577b6fff5f-7qr8c" event={"ID":"5dfe596c-b72b-4d81-a229-c0c83adb7e8e","Type":"ContainerDied","Data":"1372f5b6dc166cee8634c67bc516ef2d108c5e5382bf2f24e687997da6d816f0"}
Sep 09 21:20:24.975812 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:20:24.975811    2579 scope.go:117] "RemoveContainer" containerID="cdf7c0bc274ca398e2573dd5349448d41cbcb6e3a907e3529b89ee030921ffd9"
Sep 09 21:20:24.984855 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:20:24.984840    2579 scope.go:117] "RemoveContainer" containerID="f489cbf0f6a90f4b110caae128c2cbde865eb740882fc64eb46afe671a20771a"
Sep 09 21:20:24.992336 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:20:24.992318    2579 scope.go:117] "RemoveContainer" containerID="ea89ccd332f4653606a7438cfb6f9c8bdd3c003e7b72729207b23030a218c3b1"
Sep 09 21:20:24.998315 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:20:24.998290    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-577b6fff5f-7qr8c"]
Sep 09 21:20:25.000301 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:20:25.000276    2579 scope.go:117] "RemoveContainer" containerID="dd6ecac7e68cc8e3ac8dfcb9565ca94ba75580adbe1e6b2549d206bc969050d2"
Sep 09 21:20:25.005947 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:20:25.005924    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-577b6fff5f-7qr8c"]
Sep 09 21:20:25.008431 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:20:25.008410    2579 scope.go:117] "RemoveContainer" containerID="cdf7c0bc274ca398e2573dd5349448d41cbcb6e3a907e3529b89ee030921ffd9"
Sep 09 21:20:25.008769 ip-10-0-138-224 kubenswrapper[2579]: E0909 21:20:25.008749    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cdf7c0bc274ca398e2573dd5349448d41cbcb6e3a907e3529b89ee030921ffd9\": container with ID starting with cdf7c0bc274ca398e2573dd5349448d41cbcb6e3a907e3529b89ee030921ffd9 not found: ID does not exist" containerID="cdf7c0bc274ca398e2573dd5349448d41cbcb6e3a907e3529b89ee030921ffd9"
Sep 09 21:20:25.008847 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:20:25.008778    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cdf7c0bc274ca398e2573dd5349448d41cbcb6e3a907e3529b89ee030921ffd9"} err="failed to get container status \"cdf7c0bc274ca398e2573dd5349448d41cbcb6e3a907e3529b89ee030921ffd9\": rpc error: code = NotFound desc = could not find container \"cdf7c0bc274ca398e2573dd5349448d41cbcb6e3a907e3529b89ee030921ffd9\": container with ID starting with cdf7c0bc274ca398e2573dd5349448d41cbcb6e3a907e3529b89ee030921ffd9 not found: ID does not exist"
Sep 09 21:20:25.008847 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:20:25.008797    2579 scope.go:117] "RemoveContainer" containerID="f489cbf0f6a90f4b110caae128c2cbde865eb740882fc64eb46afe671a20771a"
Sep 09 21:20:25.009069 ip-10-0-138-224 kubenswrapper[2579]: E0909 21:20:25.009050    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f489cbf0f6a90f4b110caae128c2cbde865eb740882fc64eb46afe671a20771a\": container with ID starting with f489cbf0f6a90f4b110caae128c2cbde865eb740882fc64eb46afe671a20771a not found: ID does not exist" containerID="f489cbf0f6a90f4b110caae128c2cbde865eb740882fc64eb46afe671a20771a"
Sep 09 21:20:25.009109 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:20:25.009076    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f489cbf0f6a90f4b110caae128c2cbde865eb740882fc64eb46afe671a20771a"} err="failed to get container status \"f489cbf0f6a90f4b110caae128c2cbde865eb740882fc64eb46afe671a20771a\": rpc error: code = NotFound desc = could not find container \"f489cbf0f6a90f4b110caae128c2cbde865eb740882fc64eb46afe671a20771a\": container with ID starting with f489cbf0f6a90f4b110caae128c2cbde865eb740882fc64eb46afe671a20771a not found: ID does not exist"
Sep 09 21:20:25.009109 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:20:25.009094    2579 scope.go:117] "RemoveContainer" containerID="ea89ccd332f4653606a7438cfb6f9c8bdd3c003e7b72729207b23030a218c3b1"
Sep 09 21:20:25.009325 ip-10-0-138-224 kubenswrapper[2579]: E0909 21:20:25.009307    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea89ccd332f4653606a7438cfb6f9c8bdd3c003e7b72729207b23030a218c3b1\": container with ID starting with ea89ccd332f4653606a7438cfb6f9c8bdd3c003e7b72729207b23030a218c3b1 not found: ID does not exist" containerID="ea89ccd332f4653606a7438cfb6f9c8bdd3c003e7b72729207b23030a218c3b1"
Sep 09 21:20:25.009402 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:20:25.009334    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea89ccd332f4653606a7438cfb6f9c8bdd3c003e7b72729207b23030a218c3b1"} err="failed to get container status \"ea89ccd332f4653606a7438cfb6f9c8bdd3c003e7b72729207b23030a218c3b1\": rpc error: code = NotFound desc = could not find container \"ea89ccd332f4653606a7438cfb6f9c8bdd3c003e7b72729207b23030a218c3b1\": container with ID starting with ea89ccd332f4653606a7438cfb6f9c8bdd3c003e7b72729207b23030a218c3b1 not found: ID does not exist"
Sep 09 21:20:25.009402 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:20:25.009355    2579 scope.go:117] "RemoveContainer" containerID="dd6ecac7e68cc8e3ac8dfcb9565ca94ba75580adbe1e6b2549d206bc969050d2"
Sep 09 21:20:25.009632 ip-10-0-138-224 kubenswrapper[2579]: E0909 21:20:25.009616    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd6ecac7e68cc8e3ac8dfcb9565ca94ba75580adbe1e6b2549d206bc969050d2\": container with ID starting with dd6ecac7e68cc8e3ac8dfcb9565ca94ba75580adbe1e6b2549d206bc969050d2 not found: ID does not exist" containerID="dd6ecac7e68cc8e3ac8dfcb9565ca94ba75580adbe1e6b2549d206bc969050d2"
Sep 09 21:20:25.009673 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:20:25.009637    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd6ecac7e68cc8e3ac8dfcb9565ca94ba75580adbe1e6b2549d206bc969050d2"} err="failed to get container status \"dd6ecac7e68cc8e3ac8dfcb9565ca94ba75580adbe1e6b2549d206bc969050d2\": rpc error: code = NotFound desc = could not find container \"dd6ecac7e68cc8e3ac8dfcb9565ca94ba75580adbe1e6b2549d206bc969050d2\": container with ID starting with dd6ecac7e68cc8e3ac8dfcb9565ca94ba75580adbe1e6b2549d206bc969050d2 not found: ID does not exist"
Sep 09 21:20:25.290727 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:20:25.290694    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5dfe596c-b72b-4d81-a229-c0c83adb7e8e" path="/var/lib/kubelet/pods/5dfe596c-b72b-4d81-a229-c0c83adb7e8e/volumes"
Sep 09 21:20:27.247314 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:20:27.247282    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-wltvn/must-gather-58qnp"]
Sep 09 21:20:27.247725 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:20:27.247641    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5dfe596c-b72b-4d81-a229-c0c83adb7e8e" containerName="agent"
Sep 09 21:20:27.247725 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:20:27.247653    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="5dfe596c-b72b-4d81-a229-c0c83adb7e8e" containerName="agent"
Sep 09 21:20:27.247725 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:20:27.247676    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5dfe596c-b72b-4d81-a229-c0c83adb7e8e" containerName="storage-initializer"
Sep 09 21:20:27.247725 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:20:27.247682    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="5dfe596c-b72b-4d81-a229-c0c83adb7e8e" containerName="storage-initializer"
Sep 09 21:20:27.247725 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:20:27.247687    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5dfe596c-b72b-4d81-a229-c0c83adb7e8e" containerName="kserve-container"
Sep 09 21:20:27.247725 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:20:27.247692    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="5dfe596c-b72b-4d81-a229-c0c83adb7e8e" containerName="kserve-container"
Sep 09 21:20:27.247725 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:20:27.247703    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5dfe596c-b72b-4d81-a229-c0c83adb7e8e" containerName="kube-rbac-proxy"
Sep 09 21:20:27.247725 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:20:27.247708    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="5dfe596c-b72b-4d81-a229-c0c83adb7e8e" containerName="kube-rbac-proxy"
Sep 09 21:20:27.247981 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:20:27.247759    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="5dfe596c-b72b-4d81-a229-c0c83adb7e8e" containerName="agent"
Sep 09 21:20:27.247981 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:20:27.247778    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="5dfe596c-b72b-4d81-a229-c0c83adb7e8e" containerName="kserve-container"
Sep 09 21:20:27.247981 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:20:27.247784    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="5dfe596c-b72b-4d81-a229-c0c83adb7e8e" containerName="kube-rbac-proxy"
Sep 09 21:20:27.252590 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:20:27.252571    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-wltvn/must-gather-58qnp"
Sep 09 21:20:27.255836 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:20:27.255813    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-wltvn\"/\"kube-root-ca.crt\""
Sep 09 21:20:27.255961 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:20:27.255851    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-wltvn\"/\"openshift-service-ca.crt\""
Sep 09 21:20:27.256728 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:20:27.256706    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-wltvn\"/\"default-dockercfg-rjrx7\""
Sep 09 21:20:27.261170 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:20:27.261152    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-wltvn/must-gather-58qnp"]
Sep 09 21:20:27.312598 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:20:27.312569    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbwfm\" (UniqueName: \"kubernetes.io/projected/29079615-2674-49a3-a708-97b4789696bc-kube-api-access-wbwfm\") pod \"must-gather-58qnp\" (UID: \"29079615-2674-49a3-a708-97b4789696bc\") " pod="openshift-must-gather-wltvn/must-gather-58qnp"
Sep 09 21:20:27.312747 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:20:27.312610    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/29079615-2674-49a3-a708-97b4789696bc-must-gather-output\") pod \"must-gather-58qnp\" (UID: \"29079615-2674-49a3-a708-97b4789696bc\") " pod="openshift-must-gather-wltvn/must-gather-58qnp"
Sep 09 21:20:27.413909 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:20:27.413869    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbwfm\" (UniqueName: \"kubernetes.io/projected/29079615-2674-49a3-a708-97b4789696bc-kube-api-access-wbwfm\") pod \"must-gather-58qnp\" (UID: \"29079615-2674-49a3-a708-97b4789696bc\") " pod="openshift-must-gather-wltvn/must-gather-58qnp"
Sep 09 21:20:27.414111 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:20:27.413924    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/29079615-2674-49a3-a708-97b4789696bc-must-gather-output\") pod \"must-gather-58qnp\" (UID: \"29079615-2674-49a3-a708-97b4789696bc\") " pod="openshift-must-gather-wltvn/must-gather-58qnp"
Sep 09 21:20:27.414312 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:20:27.414289    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/29079615-2674-49a3-a708-97b4789696bc-must-gather-output\") pod \"must-gather-58qnp\" (UID: \"29079615-2674-49a3-a708-97b4789696bc\") " pod="openshift-must-gather-wltvn/must-gather-58qnp"
Sep 09 21:20:27.423003 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:20:27.422976    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbwfm\" (UniqueName: \"kubernetes.io/projected/29079615-2674-49a3-a708-97b4789696bc-kube-api-access-wbwfm\") pod \"must-gather-58qnp\" (UID: \"29079615-2674-49a3-a708-97b4789696bc\") " pod="openshift-must-gather-wltvn/must-gather-58qnp"
Sep 09 21:20:27.562149 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:20:27.562053    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-wltvn/must-gather-58qnp"
Sep 09 21:20:27.837768 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:20:27.837741    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-wltvn/must-gather-58qnp"]
Sep 09 21:20:27.840470 ip-10-0-138-224 kubenswrapper[2579]: W0909 21:20:27.840425    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod29079615_2674_49a3_a708_97b4789696bc.slice/crio-ee2a048866c52f9e658460ed4a68ed95c661c8bf59714e77f4b54abd5b1459df WatchSource:0}: Error finding container ee2a048866c52f9e658460ed4a68ed95c661c8bf59714e77f4b54abd5b1459df: Status 404 returned error can't find the container with id ee2a048866c52f9e658460ed4a68ed95c661c8bf59714e77f4b54abd5b1459df
Sep 09 21:20:27.842376 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:20:27.842355    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 21:20:27.991845 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:20:27.991813    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-wltvn/must-gather-58qnp" event={"ID":"29079615-2674-49a3-a708-97b4789696bc","Type":"ContainerStarted","Data":"ee2a048866c52f9e658460ed4a68ed95c661c8bf59714e77f4b54abd5b1459df"}
Sep 09 21:20:30.011038 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:20:30.010997    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-wltvn/must-gather-58qnp" event={"ID":"29079615-2674-49a3-a708-97b4789696bc","Type":"ContainerStarted","Data":"523db19cbea823a5dc0a2f00a487a9ed5164a51a7f1b987df4d1770b2933121b"}
Sep 09 21:20:30.011491 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:20:30.011050    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-wltvn/must-gather-58qnp" event={"ID":"29079615-2674-49a3-a708-97b4789696bc","Type":"ContainerStarted","Data":"4fe7b8b897582cfedadef0522b0b25c90f8d318ae98f3a3d706296c8fcec3572"}
Sep 09 21:20:30.028412 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:20:30.028363    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-wltvn/must-gather-58qnp" podStartSLOduration=1.317391146 podStartE2EDuration="3.028346065s" totalImagesPullingTime="1.710954919s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:20:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:20:30.026435243 +0000 UTC m=+1687.266529360" watchObservedRunningTime="2026-09-09 21:20:30.028346065 +0000 UTC m=+1687.268440182"
Sep 09 21:20:31.121049 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:20:31.121022    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-gz9zx_732d9d71-d680-4291-aaee-028307ef79dc/global-pull-secret-syncer/0.log"
Sep 09 21:20:31.241337 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:20:31.241306    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-kwvw7_9075ffbd-9633-49ac-b6aa-f410776c4ad6/konnectivity-agent/0.log"
Sep 09 21:20:31.353516 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:20:31.353454    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-138-224.ec2.internal_7e4d14d2e217a05b09faab3ef55f4d0f/haproxy/0.log"
Sep 09 21:20:35.785279 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:20:35.785238    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b3f6e9a2-a792-466e-8539-c769c5f74621/alertmanager/0.log"
Sep 09 21:20:35.818237 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:20:35.818213    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b3f6e9a2-a792-466e-8539-c769c5f74621/config-reloader/0.log"
Sep 09 21:20:35.846609 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:20:35.846574    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b3f6e9a2-a792-466e-8539-c769c5f74621/kube-rbac-proxy-web/0.log"
Sep 09 21:20:35.877426 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:20:35.877394    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b3f6e9a2-a792-466e-8539-c769c5f74621/kube-rbac-proxy/0.log"
Sep 09 21:20:35.905491 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:20:35.905246    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b3f6e9a2-a792-466e-8539-c769c5f74621/kube-rbac-proxy-metric/0.log"
Sep 09 21:20:35.937015 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:20:35.936989    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b3f6e9a2-a792-466e-8539-c769c5f74621/prom-label-proxy/0.log"
Sep 09 21:20:35.965908 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:20:35.965884    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b3f6e9a2-a792-466e-8539-c769c5f74621/init-config-reloader/0.log"
Sep 09 21:20:36.005565 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:20:36.005532    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6cdcc96965-gqh2c_d7f530ee-5bfc-4e22-acc2-328625228733/cluster-monitoring-operator/0.log"
Sep 09 21:20:36.221738 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:20:36.221708    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8lr9d_ebe524c8-2574-47dd-864c-b558de801715/node-exporter/0.log"
Sep 09 21:20:36.247464 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:20:36.247404    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8lr9d_ebe524c8-2574-47dd-864c-b558de801715/kube-rbac-proxy/0.log"
Sep 09 21:20:36.274911 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:20:36.274876    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8lr9d_ebe524c8-2574-47dd-864c-b558de801715/init-textfile/0.log"
Sep 09 21:20:36.491695 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:20:36.491616    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-84zfh_8750f8b1-ad24-45b8-b483-8bc96f4d6198/kube-rbac-proxy-main/0.log"
Sep 09 21:20:36.516433 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:20:36.516398    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-84zfh_8750f8b1-ad24-45b8-b483-8bc96f4d6198/kube-rbac-proxy-self/0.log"
Sep 09 21:20:36.543714 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:20:36.543683    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-84zfh_8750f8b1-ad24-45b8-b483-8bc96f4d6198/openshift-state-metrics/0.log"
Sep 09 21:20:37.016124 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:20:37.016087    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-8664f699df-gpz7g_7c4733c8-b3da-4ef5-92a6-b0c00921e605/telemeter-client/0.log"
Sep 09 21:20:37.079898 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:20:37.079835    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-8664f699df-gpz7g_7c4733c8-b3da-4ef5-92a6-b0c00921e605/reload/0.log"
Sep 09 21:20:37.130224 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:20:37.130190    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-8664f699df-gpz7g_7c4733c8-b3da-4ef5-92a6-b0c00921e605/kube-rbac-proxy/0.log"
Sep 09 21:20:37.181139 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:20:37.181108    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-759d4cf69c-mm926_fff59aa3-255e-4b80-94b7-2b823917c67f/thanos-query/0.log"
Sep 09 21:20:37.215431 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:20:37.215395    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-759d4cf69c-mm926_fff59aa3-255e-4b80-94b7-2b823917c67f/kube-rbac-proxy-web/0.log"
Sep 09 21:20:37.256848 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:20:37.256766    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-759d4cf69c-mm926_fff59aa3-255e-4b80-94b7-2b823917c67f/kube-rbac-proxy/0.log"
Sep 09 21:20:37.290022 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:20:37.289986    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-759d4cf69c-mm926_fff59aa3-255e-4b80-94b7-2b823917c67f/prom-label-proxy/0.log"
Sep 09 21:20:37.329028 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:20:37.328990    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-759d4cf69c-mm926_fff59aa3-255e-4b80-94b7-2b823917c67f/kube-rbac-proxy-rules/0.log"
Sep 09 21:20:37.363734 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:20:37.363687    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-759d4cf69c-mm926_fff59aa3-255e-4b80-94b7-2b823917c67f/kube-rbac-proxy-metrics/0.log"
Sep 09 21:20:39.184999 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:20:39.184959    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-wltvn/perf-node-gather-daemonset-j87hn"]
Sep 09 21:20:39.191323 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:20:39.191295    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-wltvn/perf-node-gather-daemonset-j87hn"
Sep 09 21:20:39.203615 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:20:39.203586    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-wltvn/perf-node-gather-daemonset-j87hn"]
Sep 09 21:20:39.337486 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:20:39.337432    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ecb777ec-259e-46b4-9f81-d0770e5885e7-lib-modules\") pod \"perf-node-gather-daemonset-j87hn\" (UID: \"ecb777ec-259e-46b4-9f81-d0770e5885e7\") " pod="openshift-must-gather-wltvn/perf-node-gather-daemonset-j87hn"
Sep 09 21:20:39.337486 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:20:39.337489    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9452\" (UniqueName: \"kubernetes.io/projected/ecb777ec-259e-46b4-9f81-d0770e5885e7-kube-api-access-p9452\") pod \"perf-node-gather-daemonset-j87hn\" (UID: \"ecb777ec-259e-46b4-9f81-d0770e5885e7\") " pod="openshift-must-gather-wltvn/perf-node-gather-daemonset-j87hn"
Sep 09 21:20:39.337705 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:20:39.337571    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ecb777ec-259e-46b4-9f81-d0770e5885e7-podres\") pod \"perf-node-gather-daemonset-j87hn\" (UID: \"ecb777ec-259e-46b4-9f81-d0770e5885e7\") " pod="openshift-must-gather-wltvn/perf-node-gather-daemonset-j87hn"
Sep 09 21:20:39.337705 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:20:39.337613    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ecb777ec-259e-46b4-9f81-d0770e5885e7-proc\") pod \"perf-node-gather-daemonset-j87hn\" (UID: \"ecb777ec-259e-46b4-9f81-d0770e5885e7\") " pod="openshift-must-gather-wltvn/perf-node-gather-daemonset-j87hn"
Sep 09 21:20:39.337774 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:20:39.337702    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ecb777ec-259e-46b4-9f81-d0770e5885e7-sys\") pod \"perf-node-gather-daemonset-j87hn\" (UID: \"ecb777ec-259e-46b4-9f81-d0770e5885e7\") " pod="openshift-must-gather-wltvn/perf-node-gather-daemonset-j87hn"
Sep 09 21:20:39.438477 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:20:39.438374    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ecb777ec-259e-46b4-9f81-d0770e5885e7-proc\") pod \"perf-node-gather-daemonset-j87hn\" (UID: \"ecb777ec-259e-46b4-9f81-d0770e5885e7\") " pod="openshift-must-gather-wltvn/perf-node-gather-daemonset-j87hn"
Sep 09 21:20:39.438647 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:20:39.438514    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ecb777ec-259e-46b4-9f81-d0770e5885e7-sys\") pod \"perf-node-gather-daemonset-j87hn\" (UID: \"ecb777ec-259e-46b4-9f81-d0770e5885e7\") " pod="openshift-must-gather-wltvn/perf-node-gather-daemonset-j87hn"
Sep 09 21:20:39.438647 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:20:39.438573    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ecb777ec-259e-46b4-9f81-d0770e5885e7-lib-modules\") pod \"perf-node-gather-daemonset-j87hn\" (UID: \"ecb777ec-259e-46b4-9f81-d0770e5885e7\") " pod="openshift-must-gather-wltvn/perf-node-gather-daemonset-j87hn"
Sep 09 21:20:39.438647 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:20:39.438517    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ecb777ec-259e-46b4-9f81-d0770e5885e7-proc\") pod \"perf-node-gather-daemonset-j87hn\" (UID: \"ecb777ec-259e-46b4-9f81-d0770e5885e7\") " pod="openshift-must-gather-wltvn/perf-node-gather-daemonset-j87hn"
Sep 09 21:20:39.438647 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:20:39.438598    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9452\" (UniqueName: \"kubernetes.io/projected/ecb777ec-259e-46b4-9f81-d0770e5885e7-kube-api-access-p9452\") pod \"perf-node-gather-daemonset-j87hn\" (UID: \"ecb777ec-259e-46b4-9f81-d0770e5885e7\") " pod="openshift-must-gather-wltvn/perf-node-gather-daemonset-j87hn"
Sep 09 21:20:39.438872 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:20:39.438640    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ecb777ec-259e-46b4-9f81-d0770e5885e7-sys\") pod \"perf-node-gather-daemonset-j87hn\" (UID: \"ecb777ec-259e-46b4-9f81-d0770e5885e7\") " pod="openshift-must-gather-wltvn/perf-node-gather-daemonset-j87hn"
Sep 09 21:20:39.438872 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:20:39.438672    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ecb777ec-259e-46b4-9f81-d0770e5885e7-podres\") pod \"perf-node-gather-daemonset-j87hn\" (UID: \"ecb777ec-259e-46b4-9f81-d0770e5885e7\") " pod="openshift-must-gather-wltvn/perf-node-gather-daemonset-j87hn"
Sep 09 21:20:39.438872 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:20:39.438724    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ecb777ec-259e-46b4-9f81-d0770e5885e7-lib-modules\") pod \"perf-node-gather-daemonset-j87hn\" (UID: \"ecb777ec-259e-46b4-9f81-d0770e5885e7\") " pod="openshift-must-gather-wltvn/perf-node-gather-daemonset-j87hn"
Sep 09 21:20:39.438872 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:20:39.438800    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ecb777ec-259e-46b4-9f81-d0770e5885e7-podres\") pod \"perf-node-gather-daemonset-j87hn\" (UID: \"ecb777ec-259e-46b4-9f81-d0770e5885e7\") " pod="openshift-must-gather-wltvn/perf-node-gather-daemonset-j87hn"
Sep 09 21:20:39.451719 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:20:39.451693    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9452\" (UniqueName: \"kubernetes.io/projected/ecb777ec-259e-46b4-9f81-d0770e5885e7-kube-api-access-p9452\") pod \"perf-node-gather-daemonset-j87hn\" (UID: \"ecb777ec-259e-46b4-9f81-d0770e5885e7\") " pod="openshift-must-gather-wltvn/perf-node-gather-daemonset-j87hn"
Sep 09 21:20:39.502912 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:20:39.502877    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-wltvn/perf-node-gather-daemonset-j87hn"
Sep 09 21:20:39.707733 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:20:39.707643    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-5c694d5bb7-6q8xg_746817b6-f0bd-4b14-b04e-4c30116d0045/console/0.log"
Sep 09 21:20:39.794792 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:20:39.794756    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-wltvn/perf-node-gather-daemonset-j87hn"]
Sep 09 21:20:39.801411 ip-10-0-138-224 kubenswrapper[2579]: W0909 21:20:39.801378    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podecb777ec_259e_46b4_9f81_d0770e5885e7.slice/crio-6ed43bd0529307bdc54e34b5435cd7fa0907195efdbd01539193e940dce4db6c WatchSource:0}: Error finding container 6ed43bd0529307bdc54e34b5435cd7fa0907195efdbd01539193e940dce4db6c: Status 404 returned error can't find the container with id 6ed43bd0529307bdc54e34b5435cd7fa0907195efdbd01539193e940dce4db6c
Sep 09 21:20:40.071121 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:20:40.071008    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-wltvn/perf-node-gather-daemonset-j87hn" event={"ID":"ecb777ec-259e-46b4-9f81-d0770e5885e7","Type":"ContainerStarted","Data":"c8a8e64e50e94790f32062b52a3e6f07805da2fcf2beddfb289aae26daa24836"}
Sep 09 21:20:40.071121 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:20:40.071077    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-wltvn/perf-node-gather-daemonset-j87hn" event={"ID":"ecb777ec-259e-46b4-9f81-d0770e5885e7","Type":"ContainerStarted","Data":"6ed43bd0529307bdc54e34b5435cd7fa0907195efdbd01539193e940dce4db6c"}
Sep 09 21:20:40.071337 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:20:40.071206    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-wltvn/perf-node-gather-daemonset-j87hn"
Sep 09 21:20:40.092731 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:20:40.092525    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-wltvn/perf-node-gather-daemonset-j87hn" podStartSLOduration=1.092504361 podStartE2EDuration="1.092504361s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 21:20:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:20:40.092227503 +0000 UTC m=+1697.332321623" watchObservedRunningTime="2026-09-09 21:20:40.092504361 +0000 UTC m=+1697.332598480"
Sep 09 21:20:40.344500 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:20:40.344405    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-695c895bd6-bm5qc_c31fd8ed-0c33-446d-9b0e-76fdd5935030/volume-data-source-validator/0.log"
Sep 09 21:20:41.545988 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:20:41.545956    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-zwg8k_8de0dbc6-534d-4a25-b218-8e13816e7932/dns/0.log"
Sep 09 21:20:41.572421 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:20:41.572389    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-zwg8k_8de0dbc6-534d-4a25-b218-8e13816e7932/kube-rbac-proxy/0.log"
Sep 09 21:20:41.599158 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:20:41.599132    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-88pfs_ddb26243-140d-4ac9-8a3d-4079559bd0e7/dns-node-resolver/0.log"
Sep 09 21:20:42.120024 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:20:42.119995    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-6f4bd54dd8-fqmrm_0a4dc7da-cdd3-433e-8a8f-c7726607feac/registry/0.log"
Sep 09 21:20:42.185764 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:20:42.185732    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-tml69_169cade8-7395-4b62-9544-ff0c47022a89/node-ca/0.log"
Sep 09 21:20:43.412937 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:20:43.412900    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-h4v54_5555e03e-a908-4628-a153-a8e0f4bf3f42/serve-healthcheck-canary/0.log"
Sep 09 21:20:43.957721 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:20:43.957668    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-9z7nf_8cb68ecc-5af7-4e01-8032-d41b652555ad/kube-rbac-proxy/0.log"
Sep 09 21:20:43.990243 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:20:43.990215    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-9z7nf_8cb68ecc-5af7-4e01-8032-d41b652555ad/exporter/0.log"
Sep 09 21:20:44.014885 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:20:44.014858    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-9z7nf_8cb68ecc-5af7-4e01-8032-d41b652555ad/extractor/0.log"
Sep 09 21:20:46.087056 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:20:46.087023    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-wltvn/perf-node-gather-daemonset-j87hn"
Sep 09 21:20:46.452372 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:20:46.452338    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_kserve-controller-manager-644949856f-n7kb6_45a7512f-9c23-4dd5-8507-5ccb327b29a7/manager/0.log"
Sep 09 21:20:46.652951 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:20:46.652918    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-7b966d7984-znm4k_859b8655-db7f-43c6-a775-7d06fc7a212b/manager/0.log"
Sep 09 21:20:46.749501 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:20:46.749397    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-64568bcd49-jnjbr_6918bd2d-9c6b-48b3-acca-744ea4cae141/seaweedfs/0.log"
Sep 09 21:20:51.998307 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:20:51.998276    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-fdbb8fdff-4kjtk_fd3e0c14-9e17-494f-b464-fd6b15e1f738/migrator/0.log"
Sep 09 21:20:52.031457 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:20:52.031418    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-fdbb8fdff-4kjtk_fd3e0c14-9e17-494f-b464-fd6b15e1f738/graceful-termination/0.log"
Sep 09 21:20:53.844786 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:20:53.844761    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9rprp_dfe1a979-9dff-419b-b517-93e03c19c040/kube-multus-additional-cni-plugins/0.log"
Sep 09 21:20:53.880123 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:20:53.880092    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9rprp_dfe1a979-9dff-419b-b517-93e03c19c040/egress-router-binary-copy/0.log"
Sep 09 21:20:53.913637 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:20:53.913605    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9rprp_dfe1a979-9dff-419b-b517-93e03c19c040/cni-plugins/0.log"
Sep 09 21:20:53.945727 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:20:53.945700    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9rprp_dfe1a979-9dff-419b-b517-93e03c19c040/bond-cni-plugin/0.log"
Sep 09 21:20:53.982418 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:20:53.982390    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9rprp_dfe1a979-9dff-419b-b517-93e03c19c040/routeoverride-cni/0.log"
Sep 09 21:20:54.019449 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:20:54.019422    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9rprp_dfe1a979-9dff-419b-b517-93e03c19c040/whereabouts-cni-bincopy/0.log"
Sep 09 21:20:54.055673 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:20:54.055649    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9rprp_dfe1a979-9dff-419b-b517-93e03c19c040/whereabouts-cni/0.log"
Sep 09 21:20:54.718900 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:20:54.718869    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-jbs9t_22f66b47-36ad-4226-b0f9-ad4075eaaa2d/kube-multus/0.log"
Sep 09 21:20:54.829738 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:20:54.829705    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-ws8hc_c6fd107c-ea2f-4e85-9a32-1e11f0858fd3/network-metrics-daemon/0.log"
Sep 09 21:20:54.866660 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:20:54.866634    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-ws8hc_c6fd107c-ea2f-4e85-9a32-1e11f0858fd3/kube-rbac-proxy/0.log"
Sep 09 21:20:55.931360 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:20:55.931323    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9dcqf_b2481c54-52e5-44b3-a24a-09b4dd9f575b/ovn-controller/0.log"
Sep 09 21:20:55.965738 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:20:55.965713    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9dcqf_b2481c54-52e5-44b3-a24a-09b4dd9f575b/ovn-acl-logging/0.log"
Sep 09 21:20:56.064830 ip-10-0-138-224 kubenswrapper[2579]: I0909 21:20:56.064797    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9dcqf_b2481c54-52e5-44b3-a24a-09b4dd9f575b/kube-rbac-proxy-node/0.log"