Sep 16 17:22:41.944770 ip-10-0-138-73 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 16 17:22:41.944781 ip-10-0-138-73 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 16 17:22:41.944791 ip-10-0-138-73 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 16 17:22:41.945002 ip-10-0-138-73 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 16 17:22:52.111446 ip-10-0-138-73 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 16 17:22:52.111463 ip-10-0-138-73 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 26f1b0c0826541d7bff0fa99547a3443 --
Sep 16 17:24:47.406285 ip-10-0-138-73 systemd[1]: Starting Kubernetes Kubelet...
Sep 16 17:24:47.831137 ip-10-0-138-73 kubenswrapper[2579]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 17:24:47.831137 ip-10-0-138-73 kubenswrapper[2579]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 16 17:24:47.831137 ip-10-0-138-73 kubenswrapper[2579]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 17:24:47.831137 ip-10-0-138-73 kubenswrapper[2579]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 16 17:24:47.831137 ip-10-0-138-73 kubenswrapper[2579]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 17:24:47.835643 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.835548    2579 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 16 17:24:47.839279 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.839265    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 17:24:47.839279 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.839281    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 17:24:47.839340 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.839284    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 17:24:47.839340 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.839288    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 17:24:47.839340 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.839292    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 17:24:47.839340 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.839295    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 17:24:47.839340 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.839297    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 17:24:47.839340 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.839301    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 17:24:47.839340 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.839304    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 17:24:47.839340 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.839306    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 17:24:47.839340 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.839309    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 17:24:47.839340 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.839312    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 17:24:47.839340 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.839315    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 17:24:47.839340 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.839317    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 17:24:47.839340 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.839320    2579 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 17:24:47.839340 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.839323    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 17:24:47.839340 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.839325    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 17:24:47.839340 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.839328    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 17:24:47.839340 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.839331    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 17:24:47.839340 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.839343    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 17:24:47.839340 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.839346    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 17:24:47.839340 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.839349    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 17:24:47.839818 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.839352    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 17:24:47.839818 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.839355    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 17:24:47.839818 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.839357    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 17:24:47.839818 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.839360    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 17:24:47.839818 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.839362    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 17:24:47.839818 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.839365    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 17:24:47.839818 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.839368    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 17:24:47.839818 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.839370    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 17:24:47.839818 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.839373    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 17:24:47.839818 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.839375    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 17:24:47.839818 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.839378    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 17:24:47.839818 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.839380    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 17:24:47.839818 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.839383    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 17:24:47.839818 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.839385    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 17:24:47.839818 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.839388    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 17:24:47.839818 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.839390    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 17:24:47.839818 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.839393    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 17:24:47.839818 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.839395    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 17:24:47.839818 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.839398    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 17:24:47.839818 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.839400    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 17:24:47.840327 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.839403    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 17:24:47.840327 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.839407    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 17:24:47.840327 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.839410    2579 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 17:24:47.840327 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.839413    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 17:24:47.840327 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.839415    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 17:24:47.840327 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.839418    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 17:24:47.840327 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.839421    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 17:24:47.840327 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.839423    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 17:24:47.840327 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.839426    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 17:24:47.840327 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.839430    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 17:24:47.840327 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.839435    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 17:24:47.840327 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.839438    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 17:24:47.840327 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.839441    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 17:24:47.840327 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.839444    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 17:24:47.840327 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.839447    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 17:24:47.840327 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.839450    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 17:24:47.840327 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.839454    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 17:24:47.840327 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.839457    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 17:24:47.840327 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.839460    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 17:24:47.840327 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.839462    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 17:24:47.840808 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.839466    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 17:24:47.840808 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.839469    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 17:24:47.840808 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.839471    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 17:24:47.840808 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.839474    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 17:24:47.840808 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.839477    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 17:24:47.840808 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.839480    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 17:24:47.840808 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.839482    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 17:24:47.840808 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.839486    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 17:24:47.840808 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.839489    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 17:24:47.840808 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.839491    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 17:24:47.840808 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.839494    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 17:24:47.840808 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.839497    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 17:24:47.840808 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.839499    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 17:24:47.840808 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.839502    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 17:24:47.840808 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.839505    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 17:24:47.840808 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.839508    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 17:24:47.840808 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.839511    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 17:24:47.840808 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.839515    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 17:24:47.840808 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.839519    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 17:24:47.841345 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.839522    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 17:24:47.841345 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.839524    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 17:24:47.841345 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.839527    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 17:24:47.841345 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.839530    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 17:24:47.841345 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.839532    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 17:24:47.841345 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.839536    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 17:24:47.841345 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.839539    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 17:24:47.841345 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.839542    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 17:24:47.841345 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.839544    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 17:24:47.841345 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.839547    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 17:24:47.841345 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.839549    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 17:24:47.841345 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.839552    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 17:24:47.841345 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.839555    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 17:24:47.841345 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.839557    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 17:24:47.841345 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.839560    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 17:24:47.841345 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.839564    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 17:24:47.841345 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.839567    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 17:24:47.841345 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.839570    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 17:24:47.841345 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.840271    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 17:24:47.841809 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.840278    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 17:24:47.841809 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.840281    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 17:24:47.841809 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.840286    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 17:24:47.841809 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.840290    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 17:24:47.841809 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.840293    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 17:24:47.841809 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.840296    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 17:24:47.841809 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.840299    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 17:24:47.841809 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.840302    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 17:24:47.841809 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.840306    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 17:24:47.841809 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.840309    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 17:24:47.841809 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.840311    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 17:24:47.841809 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.840316    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 17:24:47.841809 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.840319    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 17:24:47.841809 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.840322    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 17:24:47.841809 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.840325    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 17:24:47.841809 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.840329    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 17:24:47.841809 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.840332    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 17:24:47.841809 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.840335    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 17:24:47.841809 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.840339    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 17:24:47.842307 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.840343    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 17:24:47.842307 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.840345    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 17:24:47.842307 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.840348    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 17:24:47.842307 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.840351    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 17:24:47.842307 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.840353    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 17:24:47.842307 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.840356    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 17:24:47.842307 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.840359    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 17:24:47.842307 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.840362    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 17:24:47.842307 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.840364    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 17:24:47.842307 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.840367    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 17:24:47.842307 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.840370    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 17:24:47.842307 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.840372    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 17:24:47.842307 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.840375    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 17:24:47.842307 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.840377    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 17:24:47.842307 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.840380    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 17:24:47.842307 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.840383    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 17:24:47.842307 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.840385    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 17:24:47.842307 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.840388    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 17:24:47.842307 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.840390    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 17:24:47.842880 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.840393    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 17:24:47.842880 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.840396    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 17:24:47.842880 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.840399    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 17:24:47.842880 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.840402    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 17:24:47.842880 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.840405    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 17:24:47.842880 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.840407    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 17:24:47.842880 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.840410    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 17:24:47.842880 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.840413    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 17:24:47.842880 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.840415    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 17:24:47.842880 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.840418    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 17:24:47.842880 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.840421    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 17:24:47.842880 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.840423    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 17:24:47.842880 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.840426    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 17:24:47.842880 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.840430    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 17:24:47.842880 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.840432    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 17:24:47.842880 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.840435    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 17:24:47.842880 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.840438    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 17:24:47.842880 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.840440    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 17:24:47.842880 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.840443    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 17:24:47.842880 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.840445    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 17:24:47.843381 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.840448    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 17:24:47.843381 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.840451    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 17:24:47.843381 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.840454    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 17:24:47.843381 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.840456    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 17:24:47.843381 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.840459    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 17:24:47.843381 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.840462    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 17:24:47.843381 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.840465    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 17:24:47.843381 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.840468    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 17:24:47.843381 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.840471    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 17:24:47.843381 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.840473    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 17:24:47.843381 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.840476    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 17:24:47.843381 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.840479    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 17:24:47.843381 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.840481    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 17:24:47.843381 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.840484    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 17:24:47.843381 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.840487    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 17:24:47.843381 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.840489    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 17:24:47.843381 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.840492    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 17:24:47.843381 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.840496    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 17:24:47.843381 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.840499    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 17:24:47.843869 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.840502    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 17:24:47.843869 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.840505    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 17:24:47.843869 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.840507    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 17:24:47.843869 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.840510    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 17:24:47.843869 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.840513    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 17:24:47.843869 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.840516    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 17:24:47.843869 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.840518    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 17:24:47.843869 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.840521    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 17:24:47.843869 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.840524    2579 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 17:24:47.843869 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.840526    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 17:24:47.843869 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.840529    2579 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 17:24:47.843869 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.840531    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 17:24:47.843869 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.840534    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 17:24:47.843869 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.840536    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 17:24:47.843869 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.840539    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 17:24:47.843869 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.840542    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 17:24:47.843869 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.840544    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 17:24:47.843869 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.840547    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 17:24:47.843869 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.840549    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 17:24:47.843869 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.840552    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 17:24:47.843869 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.840554    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 17:24:47.844396 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.840944    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 17:24:47.844396 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.840949    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 17:24:47.844396 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.840952    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 17:24:47.844396 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.840955    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 17:24:47.844396 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.840958    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 17:24:47.844396 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.840961    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 17:24:47.844396 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.840965    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 17:24:47.844396 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.840969    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 17:24:47.844396 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.840973    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 17:24:47.844396 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.840976    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 17:24:47.844396 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.840979    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 17:24:47.844396 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.840982    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 17:24:47.844396 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.840985    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 17:24:47.844396 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.840987    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 17:24:47.844396 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.840990    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 17:24:47.844396 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.840993    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 17:24:47.844396 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.840995    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 17:24:47.844396 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.840998    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 17:24:47.844396 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.841008    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 17:24:47.844887 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.841011    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 17:24:47.844887 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.841013    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 17:24:47.844887 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.841016    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 17:24:47.844887 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.841018    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 17:24:47.844887 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.841021    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 17:24:47.844887 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.841024    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 17:24:47.844887 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.841026    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 17:24:47.844887 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.841029    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 17:24:47.844887 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.841031    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 17:24:47.844887 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.841034    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 17:24:47.844887 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.841037    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 17:24:47.844887 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.841041    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 17:24:47.844887 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.841044    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 17:24:47.844887 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.841048    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 17:24:47.844887 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.841051    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 17:24:47.844887 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.841053    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 17:24:47.844887 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.841057    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 17:24:47.844887 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.841060    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 17:24:47.844887 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.841063    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 17:24:47.845367 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.841066    2579 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 17:24:47.845367 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.841069    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 17:24:47.845367 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.841071    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 17:24:47.845367 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.841074    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 17:24:47.845367 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.841076    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 17:24:47.845367 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.841079    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 17:24:47.845367 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.841082    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 17:24:47.845367 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.841084    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 17:24:47.845367 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.841087    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 17:24:47.845367 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.841089    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 17:24:47.845367 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.841092    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 17:24:47.845367 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.841094    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 17:24:47.845367 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.841097    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 17:24:47.845367 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.841100    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 17:24:47.845367 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.841102    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 17:24:47.845367 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.841105    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 17:24:47.845367 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.841107    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 17:24:47.845367 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.841110    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 17:24:47.845367 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.841113    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 17:24:47.845367 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.841116    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 17:24:47.845888 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.841118    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 17:24:47.845888 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.841121    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 17:24:47.845888 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.841124    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 17:24:47.845888 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.841126    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 17:24:47.845888 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.841129    2579 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 17:24:47.845888 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.841131    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 17:24:47.845888 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.841134    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 17:24:47.845888 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.841137    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 17:24:47.845888 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.841139    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 17:24:47.845888 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.841142    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 17:24:47.845888 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.841144    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 17:24:47.845888 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.841147    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 17:24:47.845888 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.841150    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 17:24:47.845888 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.841152    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 17:24:47.845888 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.841155    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 17:24:47.845888 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.841157    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 17:24:47.845888 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.841160    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 17:24:47.845888 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.841162    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 17:24:47.845888 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.841165    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 17:24:47.845888 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.841167    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 17:24:47.846415 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.841170    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 17:24:47.846415 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.841172    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 17:24:47.846415 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.841175    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 17:24:47.846415 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.841177    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 17:24:47.846415 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.841180    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 17:24:47.846415 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.841182    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 17:24:47.846415 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.841185    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 17:24:47.846415 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.841188    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 17:24:47.846415 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.841191    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 17:24:47.846415 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.841195    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 17:24:47.846415 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.841201    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 17:24:47.846415 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.841204    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 17:24:47.846415 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.841207    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 17:24:47.846415 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.841209    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 17:24:47.846415 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.841212    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 17:24:47.846415 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.841215    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 17:24:47.846415 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.841217    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 17:24:47.846415 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.841220    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 17:24:47.846415 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.841222    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 17:24:47.846415 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.841225    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 17:24:47.846933 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.841227    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 17:24:47.846933 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.841303    2579 flags.go:64] FLAG: --address="0.0.0.0"
Sep 16 17:24:47.846933 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.841339    2579 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 16 17:24:47.846933 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.841345    2579 flags.go:64] FLAG: --anonymous-auth="true"
Sep 16 17:24:47.846933 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.841350    2579 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 16 17:24:47.846933 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.841355    2579 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 16 17:24:47.846933 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.841359    2579 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 16 17:24:47.846933 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.841364    2579 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 16 17:24:47.846933 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.841368    2579 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 16 17:24:47.846933 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.841371    2579 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 16 17:24:47.846933 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.841375    2579 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 16 17:24:47.846933 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.841378    2579 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 16 17:24:47.846933 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.841382    2579 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 16 17:24:47.846933 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.841385    2579 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 16 17:24:47.846933 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.841388    2579 flags.go:64] FLAG: --cgroup-root=""
Sep 16 17:24:47.846933 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.841391    2579 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 16 17:24:47.846933 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.841395    2579 flags.go:64] FLAG: --client-ca-file=""
Sep 16 17:24:47.846933 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.841397    2579 flags.go:64] FLAG: --cloud-provider="external"
Sep 16 17:24:47.846933 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.841400    2579 flags.go:64] FLAG: --cluster-dns="[]"
Sep 16 17:24:47.846933 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.841405    2579 flags.go:64] FLAG: --cluster-domain=""
Sep 16 17:24:47.846933 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.841408    2579 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 16 17:24:47.846933 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.841412    2579 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 16 17:24:47.846933 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.841417    2579 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 16 17:24:47.846933 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.841420    2579 flags.go:64] FLAG: --container-log-max-files="5"
Sep 16 17:24:47.847519 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.841424    2579 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 16 17:24:47.847519 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.841427    2579 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 16 17:24:47.847519 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.841431    2579 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 16 17:24:47.847519 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.841434    2579 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 16 17:24:47.847519 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.841438    2579 flags.go:64] FLAG: --contention-profiling="false"
Sep 16 17:24:47.847519 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.841441    2579 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 16 17:24:47.847519 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.841444    2579 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 16 17:24:47.847519 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.841448    2579 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 16 17:24:47.847519 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.841451    2579 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 16 17:24:47.847519 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.841455    2579 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 16 17:24:47.847519 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.841458    2579 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 16 17:24:47.847519 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.841461    2579 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 16 17:24:47.847519 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.841464    2579 flags.go:64] FLAG: --enable-load-reader="false"
Sep 16 17:24:47.847519 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.841467    2579 flags.go:64] FLAG: --enable-server="true"
Sep 16 17:24:47.847519 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.841470    2579 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 16 17:24:47.847519 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.841475    2579 flags.go:64] FLAG: --event-burst="100"
Sep 16 17:24:47.847519 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.841478    2579 flags.go:64] FLAG: --event-qps="50"
Sep 16 17:24:47.847519 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.841481    2579 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 16 17:24:47.847519 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.841485    2579 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 16 17:24:47.847519 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.841488    2579 flags.go:64] FLAG: --eviction-hard=""
Sep 16 17:24:47.847519 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.841491    2579 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 16 17:24:47.847519 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.841494    2579 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 16 17:24:47.847519 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.841497    2579 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 16 17:24:47.847519 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.841500    2579 flags.go:64] FLAG: --eviction-soft=""
Sep 16 17:24:47.847519 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.841503    2579 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 16 17:24:47.848134 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.841506    2579 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 16 17:24:47.848134 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.841509    2579 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 16 17:24:47.848134 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.841512    2579 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 16 17:24:47.848134 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.841516    2579 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 16 17:24:47.848134 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.841521    2579 flags.go:64] FLAG: --fail-swap-on="true"
Sep 16 17:24:47.848134 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.841525    2579 flags.go:64] FLAG: --feature-gates=""
Sep 16 17:24:47.848134 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.841533    2579 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 16 17:24:47.848134 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.841536    2579 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 16 17:24:47.848134 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.841540    2579 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 16 17:24:47.848134 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.841543    2579 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 16 17:24:47.848134 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.841546    2579 flags.go:64] FLAG: --healthz-port="10248"
Sep 16 17:24:47.848134 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.841549    2579 flags.go:64] FLAG: --help="false"
Sep 16 17:24:47.848134 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.841552    2579 flags.go:64] FLAG: --hostname-override="ip-10-0-138-73.ec2.internal"
Sep 16 17:24:47.848134 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.841556    2579 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 16 17:24:47.848134 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.841559    2579 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 16 17:24:47.848134 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.841562    2579 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 16 17:24:47.848134 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.841565    2579 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 16 17:24:47.848134 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.841569    2579 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 16 17:24:47.848134 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.841571    2579 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 16 17:24:47.848134 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.841574    2579 flags.go:64] FLAG: --image-service-endpoint=""
Sep 16 17:24:47.848134 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.841577    2579 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 16 17:24:47.848134 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.841580    2579 flags.go:64] FLAG: --kube-api-burst="100"
Sep 16 17:24:47.848134 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.841583    2579 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 16 17:24:47.848790 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.841602    2579 flags.go:64] FLAG: --kube-api-qps="50"
Sep 16 17:24:47.848790 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.841606    2579 flags.go:64] FLAG: --kube-reserved=""
Sep 16 17:24:47.848790 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.841609    2579 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 16 17:24:47.848790 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.841612    2579 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 16 17:24:47.848790 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.841616    2579 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 16 17:24:47.848790 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.841619    2579 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 16 17:24:47.848790 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.841622    2579 flags.go:64] FLAG: --lock-file=""
Sep 16 17:24:47.848790 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.841625    2579 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 16 17:24:47.848790 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.841628    2579 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 16 17:24:47.848790 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.841631    2579 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 16 17:24:47.848790 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.841636    2579 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 16 17:24:47.848790 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.841639    2579 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 16 17:24:47.848790 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.841642    2579 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 16 17:24:47.848790 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.841647    2579 flags.go:64] FLAG: --logging-format="text"
Sep 16 17:24:47.848790 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.841651    2579 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 16 17:24:47.848790 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.841655    2579 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 16 17:24:47.848790 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.841658    2579 flags.go:64] FLAG: --manifest-url=""
Sep 16 17:24:47.848790 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.841661    2579 flags.go:64] FLAG: --manifest-url-header=""
Sep 16 17:24:47.848790 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.841666    2579 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 16 17:24:47.848790 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.841669    2579 flags.go:64] FLAG: --max-open-files="1000000"
Sep 16 17:24:47.848790 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.841673    2579 flags.go:64] FLAG: --max-pods="110"
Sep 16 17:24:47.848790 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.841676    2579 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 16 17:24:47.848790 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.841679    2579 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 16 17:24:47.848790 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.841682    2579 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 16 17:24:47.848790 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.841686    2579 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 16 17:24:47.849786 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.841689    2579 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 16 17:24:47.849786 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.841692    2579 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 16 17:24:47.849786 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.841695    2579 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 16 17:24:47.849786 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.841702    2579 flags.go:64] FLAG: --node-status-max-images="50"
Sep 16 17:24:47.849786 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.841705    2579 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 16 17:24:47.849786 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.841708    2579 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 16 17:24:47.849786 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.841711    2579 flags.go:64] FLAG: --pod-cidr=""
Sep 16 17:24:47.849786 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.841715    2579 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10"
Sep 16 17:24:47.849786 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.841720    2579 flags.go:64] FLAG: --pod-manifest-path=""
Sep 16 17:24:47.849786 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.841723    2579 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 16 17:24:47.849786 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.841726    2579 flags.go:64] FLAG: --pods-per-core="0"
Sep 16 17:24:47.849786 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.841729    2579 flags.go:64] FLAG: --port="10250"
Sep 16 17:24:47.849786 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.841732    2579 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 16 17:24:47.849786 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.841735    2579 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-091a545c609572278"
Sep 16 17:24:47.849786 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.841738    2579 flags.go:64] FLAG: --qos-reserved=""
Sep 16 17:24:47.849786 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.841741    2579 flags.go:64] FLAG: --read-only-port="10255"
Sep 16 17:24:47.849786 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.841744    2579 flags.go:64] FLAG: --register-node="true"
Sep 16 17:24:47.849786 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.841747    2579 flags.go:64] FLAG: --register-with-taints=""
Sep 16 17:24:47.849786 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.841751    2579 flags.go:64] FLAG: --registry-burst="10"
Sep 16 17:24:47.849786 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.841754    2579 flags.go:64] FLAG: --registry-qps="5"
Sep 16 17:24:47.849786 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.841757    2579 flags.go:64] FLAG: --reserved-cpus=""
Sep 16 17:24:47.849786 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.841763    2579 flags.go:64] FLAG: --reserved-memory=""
Sep 16 17:24:47.849786 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.841767    2579 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 16 17:24:47.849786 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.841770    2579 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 16 17:24:47.849786 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.841773    2579 flags.go:64] FLAG: --rotate-certificates="false"
Sep 16 17:24:47.850733 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.841776    2579 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 16 17:24:47.850733 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.841779    2579 flags.go:64] FLAG: --runonce="false"
Sep 16 17:24:47.850733 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.841782    2579 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 16 17:24:47.850733 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.841786    2579 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 16 17:24:47.850733 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.841789    2579 flags.go:64] FLAG: --seccomp-default="false"
Sep 16 17:24:47.850733 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.841792    2579 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 16 17:24:47.850733 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.841795    2579 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 16 17:24:47.850733 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.841798    2579 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 16 17:24:47.850733 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.841801    2579 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 16 17:24:47.850733 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.841805    2579 flags.go:64] FLAG: --storage-driver-password="root"
Sep 16 17:24:47.850733 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.841808    2579 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 16 17:24:47.850733 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.841811    2579 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 16 17:24:47.850733 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.841814    2579 flags.go:64] FLAG: --storage-driver-user="root"
Sep 16 17:24:47.850733 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.841817    2579 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 16 17:24:47.850733 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.841821    2579 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 16 17:24:47.850733 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.841824    2579 flags.go:64] FLAG: --system-cgroups=""
Sep 16 17:24:47.850733 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.841826    2579 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 16 17:24:47.850733 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.841831    2579 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 16 17:24:47.850733 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.841834    2579 flags.go:64] FLAG: --tls-cert-file=""
Sep 16 17:24:47.850733 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.841837    2579 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 16 17:24:47.850733 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.841854    2579 flags.go:64] FLAG: --tls-min-version=""
Sep 16 17:24:47.850733 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.841857    2579 flags.go:64] FLAG: --tls-private-key-file=""
Sep 16 17:24:47.850733 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.841860    2579 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 16 17:24:47.850733 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.841863    2579 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 16 17:24:47.850733 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.841866    2579 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 16 17:24:47.851381 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.841869    2579 flags.go:64] FLAG: --v="2"
Sep 16 17:24:47.851381 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.841873    2579 flags.go:64] FLAG: --version="false"
Sep 16 17:24:47.851381 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.841877    2579 flags.go:64] FLAG: --vmodule=""
Sep 16 17:24:47.851381 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.841883    2579 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 16 17:24:47.851381 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.841887    2579 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 16 17:24:47.851381 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.842005    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 17:24:47.851381 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.842010    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 17:24:47.851381 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.842013    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 17:24:47.851381 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.842016    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 17:24:47.851381 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.842019    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 17:24:47.851381 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.842024    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 17:24:47.851381 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.842027    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 17:24:47.851381 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.842030    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 17:24:47.851381 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.842033    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 17:24:47.851381 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.842036    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 17:24:47.851381 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.842039    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 17:24:47.851381 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.842043    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 17:24:47.851381 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.842045    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 17:24:47.851381 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.842048    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 17:24:47.851381 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.842051    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 17:24:47.851905 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.842054    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 17:24:47.851905 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.842056    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 17:24:47.851905 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.842059    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 17:24:47.851905 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.842062    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 17:24:47.851905 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.842064    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 17:24:47.851905 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.842067    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 17:24:47.851905 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.842070    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 17:24:47.851905 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.842073    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 17:24:47.851905 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.842075    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 17:24:47.851905 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.842078    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 17:24:47.851905 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.842081    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 17:24:47.851905 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.842083    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 17:24:47.851905 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.842086    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 17:24:47.851905 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.842088    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 17:24:47.851905 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.842091    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 17:24:47.851905 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.842096    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 17:24:47.851905 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.842098    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 17:24:47.851905 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.842102    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 17:24:47.851905 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.842104    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 17:24:47.851905 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.842107    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 17:24:47.852400 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.842110    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 17:24:47.852400 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.842113    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 17:24:47.852400 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.842115    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 17:24:47.852400 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.842118    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 17:24:47.852400 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.842121    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 17:24:47.852400 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.842123    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 17:24:47.852400 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.842126    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 17:24:47.852400 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.842128    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 17:24:47.852400 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.842131    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 17:24:47.852400 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.842133    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 17:24:47.852400 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.842136    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 17:24:47.852400 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.842139    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 17:24:47.852400 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.842141    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 17:24:47.852400 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.842144    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 17:24:47.852400 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.842146    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 17:24:47.852400 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.842149    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 17:24:47.852400 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.842152    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 17:24:47.852400 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.842156    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 17:24:47.852400 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.842159    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 17:24:47.852902 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.842162    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 17:24:47.852902 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.842165    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 17:24:47.852902 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.842168    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 17:24:47.852902 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.842171    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 17:24:47.852902 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.842173    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 17:24:47.852902 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.842176    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 17:24:47.852902 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.842178    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 17:24:47.852902 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.842181    2579 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 17:24:47.852902 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.842185    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 17:24:47.852902 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.842188    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 17:24:47.852902 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.842191    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 17:24:47.852902 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.842195    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 17:24:47.852902 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.842198    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 17:24:47.852902 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.842201    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 17:24:47.852902 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.842204    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 17:24:47.852902 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.842206    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 17:24:47.852902 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.842209    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 17:24:47.852902 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.842211    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 17:24:47.852902 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.842214    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 17:24:47.852902 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.842217    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 17:24:47.853412 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.842220    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 17:24:47.853412 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.842222    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 17:24:47.853412 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.842225    2579 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 17:24:47.853412 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.842227    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 17:24:47.853412 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.842230    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 17:24:47.853412 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.842233    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 17:24:47.853412 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.842235    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 17:24:47.853412 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.842238    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 17:24:47.853412 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.842241    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 17:24:47.853412 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.842243    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 17:24:47.853412 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.842246    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 17:24:47.853412 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.842248    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 17:24:47.853412 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.842251    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 17:24:47.853412 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.842254    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 17:24:47.853412 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.842257    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 17:24:47.853412 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.842259    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 17:24:47.853412 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.842262    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 17:24:47.853412 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.842264    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 17:24:47.853412 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.842266    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 17:24:47.853412 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.842269    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 17:24:47.853920 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.842273    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 17:24:47.853920 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.842276    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 17:24:47.853920 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.842278    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 17:24:47.853920 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.842281    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 17:24:47.853920 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.842284    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 17:24:47.853920 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.843009    2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 17:24:47.853920 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.851915    2579 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 16 17:24:47.853920 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.852060    2579 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 16 17:24:47.853920 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.852085    2579 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 16 17:24:47.853920 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.852091    2579 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 16 17:24:47.853920 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.852150    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 17:24:47.853920 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.852155    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 17:24:47.853920 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.852159    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 17:24:47.853920 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.852162    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 17:24:47.853920 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.852166    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 17:24:47.854338 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.852169    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 17:24:47.854338 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.852172    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 17:24:47.854338 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.852174    2579 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 17:24:47.854338 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.852177    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 17:24:47.854338 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.852180    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 17:24:47.854338 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.852183    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 17:24:47.854338 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.852186    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 17:24:47.854338 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.852188    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 17:24:47.854338 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.852192    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 17:24:47.854338 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.852195    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 17:24:47.854338 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.852198    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 17:24:47.854338 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.852202    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 17:24:47.854338 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.852206    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 17:24:47.854338 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.852209    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 17:24:47.854338 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.852212    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 17:24:47.854338 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.852214    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 17:24:47.854338 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.852218    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 17:24:47.854338 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.852220    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 17:24:47.854338 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.852224    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 17:24:47.854798 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.852228    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 17:24:47.854798 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.852232    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 17:24:47.854798 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.852235    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 17:24:47.854798 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.852238    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 17:24:47.854798 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.852241    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 17:24:47.854798 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.852244    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 17:24:47.854798 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.852247    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 17:24:47.854798 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.852249    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 17:24:47.854798 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.852252    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 17:24:47.854798 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.852255    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 17:24:47.854798 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.852259    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 17:24:47.854798 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.852262    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 17:24:47.854798 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.852265    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 17:24:47.854798 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.852268    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 17:24:47.854798 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.852271    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 17:24:47.854798 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.852274    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 17:24:47.854798 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.852276    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 17:24:47.854798 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.852279    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 17:24:47.854798 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.852282    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 17:24:47.855312 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.852285    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 17:24:47.855312 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.852287    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 17:24:47.855312 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.852290    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 17:24:47.855312 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.852294    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 17:24:47.855312 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.852297    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 17:24:47.855312 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.852300    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 17:24:47.855312 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.852302    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 17:24:47.855312 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.852305    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 17:24:47.855312 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.852308    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 17:24:47.855312 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.852311    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 17:24:47.855312 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.852314    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 17:24:47.855312 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.852316    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 17:24:47.855312 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.852319    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 17:24:47.855312 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.852322    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 17:24:47.855312 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.852324    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 17:24:47.855312 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.852327    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 17:24:47.855312 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.852329    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 17:24:47.855312 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.852332    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 17:24:47.855312 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.852335    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 17:24:47.855312 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.852338    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 17:24:47.855829 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.852340    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 17:24:47.855829 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.852343    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 17:24:47.855829 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.852346    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 17:24:47.855829 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.852348    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 17:24:47.855829 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.852351    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 17:24:47.855829 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.852353    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 17:24:47.855829 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.852356    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 17:24:47.855829 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.852358    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 17:24:47.855829 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.852361    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 17:24:47.855829 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.852363    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 17:24:47.855829 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.852366    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 17:24:47.855829 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.852369    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 17:24:47.855829 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.852371    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 17:24:47.855829 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.852374    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 17:24:47.855829 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.852377    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 17:24:47.855829 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.852380    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 17:24:47.855829 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.852383    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 17:24:47.855829 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.852385    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 17:24:47.855829 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.852388    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 17:24:47.855829 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.852391    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 17:24:47.856446 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.852393    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 17:24:47.856446 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.852396    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 17:24:47.856446 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.852399    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 17:24:47.856446 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.852402    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 17:24:47.856446 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.852404    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 17:24:47.856446 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.852407    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 17:24:47.856446 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.852410    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 17:24:47.856446 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.852413    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 17:24:47.856446 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.852416    2579 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 17:24:47.856446 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.852418    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 17:24:47.856446 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.852421    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 17:24:47.856446 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.852423    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 17:24:47.856446 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.852426    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 17:24:47.856446 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.852428    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 17:24:47.856446 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.852431    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 17:24:47.856446 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.852434    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 17:24:47.856869 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.852439    2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 17:24:47.856869 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.852552    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 17:24:47.856869 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.852557    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 17:24:47.856869 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.852561    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 17:24:47.856869 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.852564    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 17:24:47.856869 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.852567    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 17:24:47.856869 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.852569    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 17:24:47.856869 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.852572    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 17:24:47.856869 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.852575    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 17:24:47.856869 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.852578    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 17:24:47.856869 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.852581    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 17:24:47.856869 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.852585    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 17:24:47.856869 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.852588    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 17:24:47.856869 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.852592    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 17:24:47.857220 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.852595    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 17:24:47.857220 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.852597    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 17:24:47.857220 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.852600    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 17:24:47.857220 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.852602    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 17:24:47.857220 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.852605    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 17:24:47.857220 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.852608    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 17:24:47.857220 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.852611    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 17:24:47.857220 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.852613    2579 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 17:24:47.857220 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.852616    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 17:24:47.857220 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.852619    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 17:24:47.857220 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.852621    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 17:24:47.857220 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.852624    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 17:24:47.857220 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.852626    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 17:24:47.857220 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.852629    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 17:24:47.857220 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.852631    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 17:24:47.857220 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.852634    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 17:24:47.857220 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.852637    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 17:24:47.857220 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.852639    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 17:24:47.857220 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.852642    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 17:24:47.857220 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.852644    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 17:24:47.857698 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.852647    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 17:24:47.857698 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.852650    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 17:24:47.857698 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.852653    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 17:24:47.857698 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.852657    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 17:24:47.857698 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.852660    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 17:24:47.857698 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.852663    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 17:24:47.857698 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.852666    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 17:24:47.857698 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.852668    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 17:24:47.857698 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.852671    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 17:24:47.857698 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.852673    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 17:24:47.857698 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.852676    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 17:24:47.857698 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.852678    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 17:24:47.857698 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.852681    2579 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 17:24:47.857698 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.852684    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 17:24:47.857698 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.852686    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 17:24:47.857698 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.852689    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 17:24:47.857698 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.852691    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 17:24:47.857698 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.852694    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 17:24:47.857698 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.852697    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 17:24:47.858183 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.852699    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 17:24:47.858183 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.852702    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 17:24:47.858183 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.852704    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 17:24:47.858183 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.852707    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 17:24:47.858183 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.852709    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 17:24:47.858183 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.852712    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 17:24:47.858183 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.852714    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 17:24:47.858183 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.852717    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 17:24:47.858183 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.852719    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 17:24:47.858183 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.852722    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 17:24:47.858183 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.852724    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 17:24:47.858183 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.852728    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 17:24:47.858183 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.852732    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 17:24:47.858183 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.852735    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 17:24:47.858183 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.852738    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 17:24:47.858183 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.852740    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 17:24:47.858183 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.852743    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 17:24:47.858183 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.852746    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 17:24:47.858183 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.852748    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 17:24:47.858796 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.852751    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 17:24:47.858796 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.852753    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 17:24:47.858796 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.852756    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 17:24:47.858796 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.852758    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 17:24:47.858796 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.852761    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 17:24:47.858796 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.852764    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 17:24:47.858796 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.852766    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 17:24:47.858796 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.852770    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 17:24:47.858796 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.852772    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 17:24:47.858796 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.852775    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 17:24:47.858796 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.852777    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 17:24:47.858796 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.852780    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 17:24:47.858796 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.852783    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 17:24:47.858796 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.852785    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 17:24:47.858796 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.852788    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 17:24:47.858796 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.852790    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 17:24:47.858796 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.852793    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 17:24:47.858796 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.852796    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 17:24:47.858796 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.852798    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 17:24:47.858796 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.852801    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 17:24:47.859315 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.852804    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 17:24:47.859315 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.852807    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 17:24:47.859315 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.852809    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 17:24:47.859315 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.852812    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 17:24:47.859315 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.852815    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 17:24:47.859315 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.852817    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 17:24:47.859315 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.852828    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 17:24:47.859315 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:47.852831    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 17:24:47.859315 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.852836    2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 17:24:47.859315 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.852966    2579 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 16 17:24:47.859315 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.854889    2579 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 16 17:24:47.859315 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.855793    2579 server.go:1019] "Starting client certificate rotation"
Sep 16 17:24:47.859315 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.855903    2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 16 17:24:47.859315 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.856675    2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 16 17:24:47.859675 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.858194    2579 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 16 17:24:47.862039 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.862019    2579 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 16 17:24:47.871118 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.871088    2579 log.go:25] "Validated CRI v1 runtime API"
Sep 16 17:24:47.874139 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.874124    2579 log.go:25] "Validated CRI v1 image API"
Sep 16 17:24:47.876330 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.876316    2579 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 16 17:24:47.880300 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.880280    2579 fs.go:135] Filesystem UUIDs: map[0a6e994e-cb04-4cfd-8aa1-6a52b04047a0:/dev/nvme0n1p4 75d86300-6b40-4fb3-8955-7f090440f61c:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2]
Sep 16 17:24:47.880368 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.880298    2579 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 16 17:24:47.886614 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.886497    2579 manager.go:216] Machine: {Timestamp:2026-09-16 17:24:47.884456821 +0000 UTC m=+0.354454948 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3098080 MemoryCapacity:33164472320 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2be5b32286f1f91c602c11a583ba7a SystemUUID:ec2be5b3-2286-f1f9-1c60-2c11a583ba7a BootID:26f1b0c0-8265-41d7-bff0-fa99547a3443 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582234112 Type:vfs Inodes:4048397 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632894464 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582238208 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:e4:b2:a0:fd Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:e4:b2:a0:fd Speed:0 Mtu:9001} {Name:ovs-system MacAddress:02:39:e3:10:ec:96 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164472320 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 16 17:24:47.886614 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.886608    2579 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 16 17:24:47.886763 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.886734    2579 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 16 17:24:47.890473 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.890439    2579 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 16 17:24:47.890619 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.890474    2579 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-138-73.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 16 17:24:47.891275 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.891264    2579 topology_manager.go:138] "Creating topology manager with none policy"
Sep 16 17:24:47.891320 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.891277    2579 container_manager_linux.go:309] "Creating device plugin manager"
Sep 16 17:24:47.891320 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.891285    2579 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 16 17:24:47.893224 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.893211    2579 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 16 17:24:47.893997 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.893987    2579 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 16 17:24:47.898362 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.898350    2579 state_mem.go:36] "Initialized new in-memory state store"
Sep 16 17:24:47.898485 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.898476    2579 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 16 17:24:47.898517 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.898508    2579 kubelet.go:486] "Attempting to sync node with API server"
Sep 16 17:24:47.898548 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.898518    2579 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 16 17:24:47.898548 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.898532    2579 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 16 17:24:47.898548 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.898542    2579 kubelet.go:398] "Adding apiserver pod source"
Sep 16 17:24:47.898632 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.898551    2579 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 16 17:24:47.899015 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.898997    2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 16 17:24:47.899015 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.899016    2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 16 17:24:47.901267 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.901246    2579 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 16 17:24:47.903374 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.903358    2579 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 16 17:24:47.903443 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.903389    2579 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 16 17:24:47.904982 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.904971    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 16 17:24:47.905020 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.904993    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 16 17:24:47.905020 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.905004    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 16 17:24:47.905020 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.905012    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 16 17:24:47.905121 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.905025    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 16 17:24:47.905121 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.905034    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 16 17:24:47.905121 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.905056    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 16 17:24:47.905121 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.905064    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 16 17:24:47.905121 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.905072    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 16 17:24:47.905121 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.905079    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 16 17:24:47.905121 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.905091    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 16 17:24:47.905121 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.905098    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 16 17:24:47.905348 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.905131    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 16 17:24:47.905348 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.905137    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 16 17:24:47.906023 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.906007    2579 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 16 17:24:47.909209 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.909196    2579 server.go:1268] "Started kubelet"
Sep 16 17:24:47.909310 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.909286    2579 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 16 17:24:47.909405 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.909359    2579 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 16 17:24:47.909441 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.909424    2579 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 16 17:24:47.910656 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.910634    2579 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 16 17:24:47.910893 ip-10-0-138-73 systemd[1]: Started Kubernetes Kubelet.
Sep 16 17:24:47.911776 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.911761    2579 server.go:310] "Adding debug handlers to kubelet server"
Sep 16 17:24:47.914323 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:24:47.914261    2579 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 16 17:24:47.914419 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.914326    2579 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-73.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 16 17:24:47.914468 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:24:47.914450    2579 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-138-73.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 16 17:24:47.914812 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:24:47.914797    2579 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 16 17:24:47.916279 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.916260    2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 16 17:24:47.916895 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.916876    2579 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 16 17:24:47.917230 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:24:47.917191    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-73.ec2.internal\" not found"
Sep 16 17:24:47.917607 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.917566    2579 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 16 17:24:47.917607 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.917592    2579 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 16 17:24:47.917741 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.917671    2579 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 16 17:24:47.917794 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.917767    2579 reconstruct.go:97] "Volume reconstruction finished"
Sep 16 17:24:47.917794 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.917774    2579 reconciler.go:29] "Reconciler: start to sync state"
Sep 16 17:24:47.917963 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.917949    2579 factory.go:55] Registering systemd factory
Sep 16 17:24:47.918016 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.917990    2579 factory.go:223] Registration of the systemd container factory successfully
Sep 16 17:24:47.918234 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.918217    2579 factory.go:153] Registering CRI-O factory
Sep 16 17:24:47.918234 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.918235    2579 factory.go:223] Registration of the crio container factory successfully
Sep 16 17:24:47.918375 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.918278    2579 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 16 17:24:47.918375 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.918292    2579 factory.go:103] Registering Raw factory
Sep 16 17:24:47.918375 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.918303    2579 manager.go:1196] Started watching for new ooms in manager
Sep 16 17:24:47.918599 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.918588    2579 manager.go:349] Starting recovery of all containers
Sep 16 17:24:47.930879 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.930838    2579 manager.go:354] Recovery completed
Sep 16 17:24:47.931376 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:24:47.931356    2579 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 16 17:24:47.931609 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:24:47.931584    2579 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-138-73.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 16 17:24:47.932274 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:24:47.931310    2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-73.ec2.internal.18d5dd100a26c7f9  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-73.ec2.internal,UID:ip-10-0-138-73.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-138-73.ec2.internal,},FirstTimestamp:2026-09-16 17:24:47.909169145 +0000 UTC m=+0.379167271,LastTimestamp:2026-09-16 17:24:47.909169145 +0000 UTC m=+0.379167271,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-73.ec2.internal,}"
Sep 16 17:24:47.935677 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.935663    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 17:24:47.937229 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.937213    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-73.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 17:24:47.937316 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.937243    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-73.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 17:24:47.937316 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.937255    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-73.ec2.internal" event="NodeHasSufficientPID"
Sep 16 17:24:47.937887 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.937872    2579 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 16 17:24:47.937887 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.937883    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 17:24:47.938020 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.937886    2579 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 16 17:24:47.938020 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.937921    2579 state_mem.go:36] "Initialized new in-memory state store"
Sep 16 17:24:47.938588 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.938571    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-73.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 17:24:47.938662 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.938600    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-73.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 17:24:47.938662 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.938610    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-73.ec2.internal" event="NodeHasSufficientPID"
Sep 16 17:24:47.939196 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.939184    2579 policy_none.go:49] "None policy: Start"
Sep 16 17:24:47.939244 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.939201    2579 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 16 17:24:47.939244 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.939212    2579 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 16 17:24:47.940131 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.940116    2579 policy_none.go:47] "Start"
Sep 16 17:24:47.941904 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:24:47.941811    2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-73.ec2.internal.18d5dd100bd2ed5d  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-73.ec2.internal,UID:ip-10-0-138-73.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-138-73.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-138-73.ec2.internal,},FirstTimestamp:2026-09-16 17:24:47.937228125 +0000 UTC m=+0.407226251,LastTimestamp:2026-09-16 17:24:47.937228125 +0000 UTC m=+0.407226251,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-73.ec2.internal,}"
Sep 16 17:24:47.955300 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:24:47.955226    2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-73.ec2.internal.18d5dd100bd3413b  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-73.ec2.internal,UID:ip-10-0-138-73.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-138-73.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-138-73.ec2.internal,},FirstTimestamp:2026-09-16 17:24:47.937249595 +0000 UTC m=+0.407247720,LastTimestamp:2026-09-16 17:24:47.937249595 +0000 UTC m=+0.407247720,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-73.ec2.internal,}"
Sep 16 17:24:47.968435 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.968419    2579 manager.go:337] "Starting Device Plugin manager"
Sep 16 17:24:48.090105 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:24:47.968469    2579 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 16 17:24:48.090105 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.968482    2579 server.go:87] "Starting device plugin registration server"
Sep 16 17:24:48.090105 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.968772    2579 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 16 17:24:48.090105 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.968784    2579 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 16 17:24:48.090105 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.968968    2579 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 16 17:24:48.090105 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.969032    2579 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 16 17:24:48.090105 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:47.969040    2579 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 16 17:24:48.090105 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:24:47.969608    2579 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 16 17:24:48.090105 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:24:47.969652    2579 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-138-73.ec2.internal\" not found"
Sep 16 17:24:48.090105 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:24:47.976021    2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-73.ec2.internal.18d5dd100bd368be  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-73.ec2.internal,UID:ip-10-0-138-73.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-138-73.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-138-73.ec2.internal,},FirstTimestamp:2026-09-16 17:24:47.93725971 +0000 UTC m=+0.407257836,LastTimestamp:2026-09-16 17:24:47.93725971 +0000 UTC m=+0.407257836,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-73.ec2.internal,}"
Sep 16 17:24:48.090105 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:48.000984    2579 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-lgsxd"
Sep 16 17:24:48.090105 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:24:48.010859    2579 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-138-73.ec2.internal.18d5dd100bd2ed5d\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-73.ec2.internal.18d5dd100bd2ed5d  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-73.ec2.internal,UID:ip-10-0-138-73.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-138-73.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-138-73.ec2.internal,},FirstTimestamp:2026-09-16 17:24:47.937228125 +0000 UTC m=+0.407226251,LastTimestamp:2026-09-16 17:24:47.938586999 +0000 UTC m=+0.408585127,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-73.ec2.internal,}"
Sep 16 17:24:48.090105 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:48.018391    2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-lgsxd"
Sep 16 17:24:48.090105 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:24:48.024194    2579 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-138-73.ec2.internal.18d5dd100bd3413b\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-73.ec2.internal.18d5dd100bd3413b  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-73.ec2.internal,UID:ip-10-0-138-73.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-138-73.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-138-73.ec2.internal,},FirstTimestamp:2026-09-16 17:24:47.937249595 +0000 UTC m=+0.407247720,LastTimestamp:2026-09-16 17:24:47.938605125 +0000 UTC m=+0.408603251,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-73.ec2.internal,}"
Sep 16 17:24:48.090105 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:48.069914    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 17:24:48.090105 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:48.070918    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-73.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 17:24:48.090105 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:48.070951    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-73.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 17:24:48.090105 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:48.070961    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-73.ec2.internal" event="NodeHasSufficientPID"
Sep 16 17:24:48.090105 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:48.070991    2579 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-138-73.ec2.internal"
Sep 16 17:24:48.090105 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:48.089734    2579 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-138-73.ec2.internal"
Sep 16 17:24:48.090105 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:24:48.089758    2579 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-138-73.ec2.internal\": node \"ip-10-0-138-73.ec2.internal\" not found"
Sep 16 17:24:48.091578 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:48.091555    2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 16 17:24:48.092838 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:48.092813    2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 16 17:24:48.092948 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:48.092879    2579 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 16 17:24:48.092948 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:48.092902    2579 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 16 17:24:48.092948 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:24:48.092940    2579 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 16 17:24:48.101605 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:48.101586    2579 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 17:24:48.149087 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:24:48.149057    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-73.ec2.internal\" not found"
Sep 16 17:24:48.193605 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:48.193569    2579 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-73.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-138-73.ec2.internal"]
Sep 16 17:24:48.193769 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:48.193649    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 17:24:48.194694 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:48.194670    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-73.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 17:24:48.194815 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:48.194707    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-73.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 17:24:48.194815 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:48.194721    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-73.ec2.internal" event="NodeHasSufficientPID"
Sep 16 17:24:48.196840 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:48.196824    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 17:24:48.196942 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:48.196932    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-73.ec2.internal"
Sep 16 17:24:48.196990 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:48.196962    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 17:24:48.197727 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:48.197706    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-73.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 17:24:48.197811 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:48.197736    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-73.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 17:24:48.197811 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:48.197756    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-73.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 17:24:48.197811 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:48.197770    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-73.ec2.internal" event="NodeHasSufficientPID"
Sep 16 17:24:48.197811 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:48.197739    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-73.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 17:24:48.197978 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:48.197826    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-73.ec2.internal" event="NodeHasSufficientPID"
Sep 16 17:24:48.199675 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:48.199661    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-73.ec2.internal"
Sep 16 17:24:48.199723 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:48.199684    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 17:24:48.200398 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:48.200382    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-73.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 17:24:48.200449 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:48.200413    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-73.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 17:24:48.200449 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:48.200428    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-73.ec2.internal" event="NodeHasSufficientPID"
Sep 16 17:24:48.215379 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:24:48.215358    2579 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-73.ec2.internal\" not found" node="ip-10-0-138-73.ec2.internal"
Sep 16 17:24:48.218656 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:24:48.218641    2579 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-73.ec2.internal\" not found" node="ip-10-0-138-73.ec2.internal"
Sep 16 17:24:48.250028 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:24:48.249991    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-73.ec2.internal\" not found"
Sep 16 17:24:48.318806 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:48.318779    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ffbb989cf9bf70887695397b1f4835f5-config\") pod \"kube-apiserver-proxy-ip-10-0-138-73.ec2.internal\" (UID: \"ffbb989cf9bf70887695397b1f4835f5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-73.ec2.internal"
Sep 16 17:24:48.318806 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:48.318809    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/14dbc6cf811b53cc1d9f6ac780fa88ac-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-73.ec2.internal\" (UID: \"14dbc6cf811b53cc1d9f6ac780fa88ac\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-73.ec2.internal"
Sep 16 17:24:48.318988 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:48.318825    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/14dbc6cf811b53cc1d9f6ac780fa88ac-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-73.ec2.internal\" (UID: \"14dbc6cf811b53cc1d9f6ac780fa88ac\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-73.ec2.internal"
Sep 16 17:24:48.350958 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:24:48.350885    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-73.ec2.internal\" not found"
Sep 16 17:24:48.419465 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:48.419437    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/14dbc6cf811b53cc1d9f6ac780fa88ac-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-73.ec2.internal\" (UID: \"14dbc6cf811b53cc1d9f6ac780fa88ac\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-73.ec2.internal"
Sep 16 17:24:48.419562 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:48.419479    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/14dbc6cf811b53cc1d9f6ac780fa88ac-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-73.ec2.internal\" (UID: \"14dbc6cf811b53cc1d9f6ac780fa88ac\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-73.ec2.internal"
Sep 16 17:24:48.419562 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:48.419495    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ffbb989cf9bf70887695397b1f4835f5-config\") pod \"kube-apiserver-proxy-ip-10-0-138-73.ec2.internal\" (UID: \"ffbb989cf9bf70887695397b1f4835f5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-73.ec2.internal"
Sep 16 17:24:48.419562 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:48.419517    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ffbb989cf9bf70887695397b1f4835f5-config\") pod \"kube-apiserver-proxy-ip-10-0-138-73.ec2.internal\" (UID: \"ffbb989cf9bf70887695397b1f4835f5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-73.ec2.internal"
Sep 16 17:24:48.419562 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:48.419439    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/14dbc6cf811b53cc1d9f6ac780fa88ac-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-73.ec2.internal\" (UID: \"14dbc6cf811b53cc1d9f6ac780fa88ac\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-73.ec2.internal"
Sep 16 17:24:48.419677 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:48.419562    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/14dbc6cf811b53cc1d9f6ac780fa88ac-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-73.ec2.internal\" (UID: \"14dbc6cf811b53cc1d9f6ac780fa88ac\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-73.ec2.internal"
Sep 16 17:24:48.451293 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:24:48.451267    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-73.ec2.internal\" not found"
Sep 16 17:24:48.517949 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:48.517916    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-73.ec2.internal"
Sep 16 17:24:48.521132 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:48.521114    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-73.ec2.internal"
Sep 16 17:24:48.552236 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:24:48.552204    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-73.ec2.internal\" not found"
Sep 16 17:24:48.653308 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:24:48.653222    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-73.ec2.internal\" not found"
Sep 16 17:24:48.753494 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:24:48.753459    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-73.ec2.internal\" not found"
Sep 16 17:24:48.805563 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:48.805536    2579 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 17:24:48.820751 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:48.820726    2579 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 17:24:48.854172 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:24:48.854142    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-73.ec2.internal\" not found"
Sep 16 17:24:48.856305 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:48.856287    2579 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 16 17:24:48.856437 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:48.856420    2579 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 16 17:24:48.856473 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:48.856451    2579 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 16 17:24:48.856502 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:48.856452    2579 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 16 17:24:48.920790 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:48.920724    2579 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 17:24:49.018888 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:49.018653    2579 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-73.ec2.internal"
Sep 16 17:24:49.021826 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:49.021794    2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-15 17:19:47 +0000 UTC" deadline="2028-02-10 09:46:55.237502908 +0000 UTC"
Sep 16 17:24:49.021826 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:49.021822    2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12280h22m6.215683884s"
Sep 16 17:24:49.040047 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:49.040019    2579 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 16 17:24:49.040808 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:49.040794    2579 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-73.ec2.internal"
Sep 16 17:24:49.070800 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:49.070765    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffbb989cf9bf70887695397b1f4835f5.slice/crio-28bc02505749d756d415deec59c0a4607f6520566997605d31b684ba4dfcdb09 WatchSource:0}: Error finding container 28bc02505749d756d415deec59c0a4607f6520566997605d31b684ba4dfcdb09: Status 404 returned error can't find the container with id 28bc02505749d756d415deec59c0a4607f6520566997605d31b684ba4dfcdb09
Sep 16 17:24:49.071176 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:49.071147    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14dbc6cf811b53cc1d9f6ac780fa88ac.slice/crio-7ef54a0623248a148f0ddf99279c3aff90a4d8798025871998a886a0dd5b4c54 WatchSource:0}: Error finding container 7ef54a0623248a148f0ddf99279c3aff90a4d8798025871998a886a0dd5b4c54: Status 404 returned error can't find the container with id 7ef54a0623248a148f0ddf99279c3aff90a4d8798025871998a886a0dd5b4c54
Sep 16 17:24:49.076026 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:49.076010    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 17:24:49.076789 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:49.076704    2579 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 16 17:24:49.095935 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:49.095881    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-73.ec2.internal" event={"ID":"14dbc6cf811b53cc1d9f6ac780fa88ac","Type":"ContainerStarted","Data":"7ef54a0623248a148f0ddf99279c3aff90a4d8798025871998a886a0dd5b4c54"}
Sep 16 17:24:49.096888 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:49.096869    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-73.ec2.internal" event={"ID":"ffbb989cf9bf70887695397b1f4835f5","Type":"ContainerStarted","Data":"28bc02505749d756d415deec59c0a4607f6520566997605d31b684ba4dfcdb09"}
Sep 16 17:24:49.795054 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:49.795024    2579 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 17:24:49.899375 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:49.899342    2579 apiserver.go:52] "Watching apiserver"
Sep 16 17:24:49.909272 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:49.909244    2579 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 16 17:24:49.909742 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:49.909719    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-138-73.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w58mp","openshift-dns/node-resolver-qtdvz","openshift-multus/multus-59rw5","openshift-multus/multus-additional-cni-plugins-6fsn7","openshift-multus/network-metrics-daemon-xfgjq","kube-system/global-pull-secret-syncer-s7zg6","openshift-cluster-node-tuning-operator/tuned-pgq7f","openshift-image-registry/node-ca-t6cx6","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-73.ec2.internal","openshift-network-diagnostics/network-check-target-4c79x","openshift-network-operator/iptables-alerter-gmmwp","openshift-ovn-kubernetes/ovnkube-node-f9qxl","kube-system/konnectivity-agent-gntnz"]
Sep 16 17:24:49.912320 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:49.912296    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-gmmwp"
Sep 16 17:24:49.914557 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:49.914511    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w58mp"
Sep 16 17:24:49.916410 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:49.916388    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-gntnz"
Sep 16 17:24:49.916688 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:49.916594    2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 16 17:24:49.917053 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:49.917028    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 16 17:24:49.917500 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:49.917474    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 16 17:24:49.917500 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:49.917485    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 16 17:24:49.917786 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:49.917753    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-7gtfd\""
Sep 16 17:24:49.918435 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:49.918390    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-59rw5"
Sep 16 17:24:49.920719 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:49.920702    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s7zg6"
Sep 16 17:24:49.920830 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:24:49.920806    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s7zg6" podUID="e9240885-14d4-4b43-bf98-442e30d61c36"
Sep 16 17:24:49.922479 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:49.922460    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xfgjq"
Sep 16 17:24:49.922566 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:24:49.922531    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xfgjq" podUID="bc719829-66b1-4a84-b994-e71cf111dafa"
Sep 16 17:24:49.924521 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:49.924502    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-pgq7f"
Sep 16 17:24:49.926136 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:49.925973    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 16 17:24:49.926136 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:49.926041    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/1ea5d21d-1046-4edb-b1d9-9ca43ebc564d-iptables-alerter-script\") pod \"iptables-alerter-gmmwp\" (UID: \"1ea5d21d-1046-4edb-b1d9-9ca43ebc564d\") " pod="openshift-network-operator/iptables-alerter-gmmwp"
Sep 16 17:24:49.926136 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:49.926070    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vkjgp\" (UniqueName: \"kubernetes.io/projected/1ea5d21d-1046-4edb-b1d9-9ca43ebc564d-kube-api-access-vkjgp\") pod \"iptables-alerter-gmmwp\" (UID: \"1ea5d21d-1046-4edb-b1d9-9ca43ebc564d\") " pod="openshift-network-operator/iptables-alerter-gmmwp"
Sep 16 17:24:49.926136 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:49.926094    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a1223407-0c9e-4ed6-bd17-cf0f3962e27e-etc-kubernetes\") pod \"multus-59rw5\" (UID: \"a1223407-0c9e-4ed6-bd17-cf0f3962e27e\") " pod="openshift-multus/multus-59rw5"
Sep 16 17:24:49.926136 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:49.926115    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a1223407-0c9e-4ed6-bd17-cf0f3962e27e-cnibin\") pod \"multus-59rw5\" (UID: \"a1223407-0c9e-4ed6-bd17-cf0f3962e27e\") " pod="openshift-multus/multus-59rw5"
Sep 16 17:24:49.926466 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:49.926144    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a1223407-0c9e-4ed6-bd17-cf0f3962e27e-host-run-k8s-cni-cncf-io\") pod \"multus-59rw5\" (UID: \"a1223407-0c9e-4ed6-bd17-cf0f3962e27e\") " pod="openshift-multus/multus-59rw5"
Sep 16 17:24:49.926466 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:49.926164    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a1223407-0c9e-4ed6-bd17-cf0f3962e27e-host-run-multus-certs\") pod \"multus-59rw5\" (UID: \"a1223407-0c9e-4ed6-bd17-cf0f3962e27e\") " pod="openshift-multus/multus-59rw5"
Sep 16 17:24:49.926466 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:49.926184    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1ea5d21d-1046-4edb-b1d9-9ca43ebc564d-host-slash\") pod \"iptables-alerter-gmmwp\" (UID: \"1ea5d21d-1046-4edb-b1d9-9ca43ebc564d\") " pod="openshift-network-operator/iptables-alerter-gmmwp"
Sep 16 17:24:49.926466 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:49.926203    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a1223407-0c9e-4ed6-bd17-cf0f3962e27e-cni-binary-copy\") pod \"multus-59rw5\" (UID: \"a1223407-0c9e-4ed6-bd17-cf0f3962e27e\") " pod="openshift-multus/multus-59rw5"
Sep 16 17:24:49.926466 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:49.926246    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a1223407-0c9e-4ed6-bd17-cf0f3962e27e-host-run-netns\") pod \"multus-59rw5\" (UID: \"a1223407-0c9e-4ed6-bd17-cf0f3962e27e\") " pod="openshift-multus/multus-59rw5"
Sep 16 17:24:49.926466 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:49.926297    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvjm2\" (UniqueName: \"kubernetes.io/projected/a1223407-0c9e-4ed6-bd17-cf0f3962e27e-kube-api-access-nvjm2\") pod \"multus-59rw5\" (UID: \"a1223407-0c9e-4ed6-bd17-cf0f3962e27e\") " pod="openshift-multus/multus-59rw5"
Sep 16 17:24:49.926466 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:49.926336    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c73a8482-2abb-407e-972d-05a54828be7a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-w58mp\" (UID: \"c73a8482-2abb-407e-972d-05a54828be7a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w58mp"
Sep 16 17:24:49.926466 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:49.926359    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c73a8482-2abb-407e-972d-05a54828be7a-device-dir\") pod \"aws-ebs-csi-driver-node-w58mp\" (UID: \"c73a8482-2abb-407e-972d-05a54828be7a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w58mp"
Sep 16 17:24:49.926466 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:49.926381    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a1223407-0c9e-4ed6-bd17-cf0f3962e27e-multus-cni-dir\") pod \"multus-59rw5\" (UID: \"a1223407-0c9e-4ed6-bd17-cf0f3962e27e\") " pod="openshift-multus/multus-59rw5"
Sep 16 17:24:49.926466 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:49.926403    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a1223407-0c9e-4ed6-bd17-cf0f3962e27e-host-var-lib-cni-multus\") pod \"multus-59rw5\" (UID: \"a1223407-0c9e-4ed6-bd17-cf0f3962e27e\") " pod="openshift-multus/multus-59rw5"
Sep 16 17:24:49.926466 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:49.926428    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c73a8482-2abb-407e-972d-05a54828be7a-registration-dir\") pod \"aws-ebs-csi-driver-node-w58mp\" (UID: \"c73a8482-2abb-407e-972d-05a54828be7a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w58mp"
Sep 16 17:24:49.926466 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:49.926458    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c73a8482-2abb-407e-972d-05a54828be7a-etc-selinux\") pod \"aws-ebs-csi-driver-node-w58mp\" (UID: \"c73a8482-2abb-407e-972d-05a54828be7a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w58mp"
Sep 16 17:24:49.927010 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:49.926488    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c73a8482-2abb-407e-972d-05a54828be7a-sys-fs\") pod \"aws-ebs-csi-driver-node-w58mp\" (UID: \"c73a8482-2abb-407e-972d-05a54828be7a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w58mp"
Sep 16 17:24:49.927010 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:49.926521    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ef27aa30-cb45-45bc-8d0e-50107fc6459d-agent-certs\") pod \"konnectivity-agent-gntnz\" (UID: \"ef27aa30-cb45-45bc-8d0e-50107fc6459d\") " pod="kube-system/konnectivity-agent-gntnz"
Sep 16 17:24:49.927010 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:49.926559    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a1223407-0c9e-4ed6-bd17-cf0f3962e27e-system-cni-dir\") pod \"multus-59rw5\" (UID: \"a1223407-0c9e-4ed6-bd17-cf0f3962e27e\") " pod="openshift-multus/multus-59rw5"
Sep 16 17:24:49.927010 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:49.926581    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vbr5\" (UniqueName: \"kubernetes.io/projected/c73a8482-2abb-407e-972d-05a54828be7a-kube-api-access-6vbr5\") pod \"aws-ebs-csi-driver-node-w58mp\" (UID: \"c73a8482-2abb-407e-972d-05a54828be7a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w58mp"
Sep 16 17:24:49.927010 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:49.926605    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ef27aa30-cb45-45bc-8d0e-50107fc6459d-konnectivity-ca\") pod \"konnectivity-agent-gntnz\" (UID: \"ef27aa30-cb45-45bc-8d0e-50107fc6459d\") " pod="kube-system/konnectivity-agent-gntnz"
Sep 16 17:24:49.927010 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:49.926613    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-t6cx6"
Sep 16 17:24:49.927010 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:49.926627    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a1223407-0c9e-4ed6-bd17-cf0f3962e27e-host-var-lib-kubelet\") pod \"multus-59rw5\" (UID: \"a1223407-0c9e-4ed6-bd17-cf0f3962e27e\") " pod="openshift-multus/multus-59rw5"
Sep 16 17:24:49.927010 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:49.926662    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a1223407-0c9e-4ed6-bd17-cf0f3962e27e-multus-conf-dir\") pod \"multus-59rw5\" (UID: \"a1223407-0c9e-4ed6-bd17-cf0f3962e27e\") " pod="openshift-multus/multus-59rw5"
Sep 16 17:24:49.927010 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:49.926685    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a1223407-0c9e-4ed6-bd17-cf0f3962e27e-multus-daemon-config\") pod \"multus-59rw5\" (UID: \"a1223407-0c9e-4ed6-bd17-cf0f3962e27e\") " pod="openshift-multus/multus-59rw5"
Sep 16 17:24:49.927010 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:49.926705    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c73a8482-2abb-407e-972d-05a54828be7a-socket-dir\") pod \"aws-ebs-csi-driver-node-w58mp\" (UID: \"c73a8482-2abb-407e-972d-05a54828be7a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w58mp"
Sep 16 17:24:49.927010 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:49.926726    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a1223407-0c9e-4ed6-bd17-cf0f3962e27e-host-var-lib-cni-bin\") pod \"multus-59rw5\" (UID: \"a1223407-0c9e-4ed6-bd17-cf0f3962e27e\") " pod="openshift-multus/multus-59rw5"
Sep 16 17:24:49.927010 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:49.926746    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a1223407-0c9e-4ed6-bd17-cf0f3962e27e-hostroot\") pod \"multus-59rw5\" (UID: \"a1223407-0c9e-4ed6-bd17-cf0f3962e27e\") " pod="openshift-multus/multus-59rw5"
Sep 16 17:24:49.927010 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:49.926775    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a1223407-0c9e-4ed6-bd17-cf0f3962e27e-multus-socket-dir-parent\") pod \"multus-59rw5\" (UID: \"a1223407-0c9e-4ed6-bd17-cf0f3962e27e\") " pod="openshift-multus/multus-59rw5"
Sep 16 17:24:49.929125 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:49.929105    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4c79x"
Sep 16 17:24:49.929217 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:24:49.929196    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4c79x" podUID="1d620b9d-0055-4ed6-aced-2f4cfc54cbd8"
Sep 16 17:24:49.931305 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:49.931285    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-6fsn7"
Sep 16 17:24:49.931739 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:49.931724    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 16 17:24:49.932531 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:49.932446    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 16 17:24:49.933152 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:49.933124    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 16 17:24:49.934755 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:49.933421    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 16 17:24:49.934755 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:49.934417    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 16 17:24:49.934899 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:49.934879    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 16 17:24:49.935205 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:49.935187    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 16 17:24:49.935419 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:49.935398    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 16 17:24:49.935471 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:49.935433    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-82pvq\""
Sep 16 17:24:49.936183 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:49.936157    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-f9qxl"
Sep 16 17:24:49.939244 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:49.939226    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-qtdvz"
Sep 16 17:24:49.948390 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:49.948282    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 16 17:24:49.948502 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:49.948433    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-mpxnr\""
Sep 16 17:24:49.948808 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:49.948696    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 16 17:24:49.948808 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:49.948714    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 16 17:24:49.948808 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:49.948776    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 16 17:24:49.949267 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:49.949247    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 16 17:24:49.949710 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:49.949682    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 16 17:24:49.949710 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:49.949693    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 16 17:24:49.950211 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:49.949940    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 16 17:24:49.950211 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:49.949972    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 16 17:24:49.950211 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:49.949940    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 16 17:24:49.950211 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:49.950174    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 16 17:24:49.951427 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:49.950515    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-4tk98\""
Sep 16 17:24:49.951427 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:49.950715    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-pvqfb\""
Sep 16 17:24:49.951427 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:49.950957    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 16 17:24:49.951427 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:49.951087    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 16 17:24:49.951427 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:49.951144    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 16 17:24:49.951427 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:49.951311    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-x99mg\""
Sep 16 17:24:49.951427 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:49.951335    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 16 17:24:49.951867 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:49.951504    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-pbxrr\""
Sep 16 17:24:49.951867 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:49.951664    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-kbrdz\""
Sep 16 17:24:49.957859 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:49.957105    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-mgz9t\""
Sep 16 17:24:49.981356 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:49.981323    2579 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 16 17:24:50.019701 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:50.019674    2579 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 16 17:24:50.026109 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:50.026075    2579 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-4xl76"
Sep 16 17:24:50.026992 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:50.026967    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a1223407-0c9e-4ed6-bd17-cf0f3962e27e-multus-conf-dir\") pod \"multus-59rw5\" (UID: \"a1223407-0c9e-4ed6-bd17-cf0f3962e27e\") " pod="openshift-multus/multus-59rw5"
Sep 16 17:24:50.027114 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:50.027008    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c2617cf7-6369-4fc4-9fb5-6e6cfc4e9add-systemd-units\") pod \"ovnkube-node-f9qxl\" (UID: \"c2617cf7-6369-4fc4-9fb5-6e6cfc4e9add\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9qxl"
Sep 16 17:24:50.027114 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:50.027030    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c2617cf7-6369-4fc4-9fb5-6e6cfc4e9add-node-log\") pod \"ovnkube-node-f9qxl\" (UID: \"c2617cf7-6369-4fc4-9fb5-6e6cfc4e9add\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9qxl"
Sep 16 17:24:50.027114 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:50.027055    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c2617cf7-6369-4fc4-9fb5-6e6cfc4e9add-env-overrides\") pod \"ovnkube-node-f9qxl\" (UID: \"c2617cf7-6369-4fc4-9fb5-6e6cfc4e9add\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9qxl"
Sep 16 17:24:50.027114 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:50.027078    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmzzm\" (UniqueName: \"kubernetes.io/projected/bc719829-66b1-4a84-b994-e71cf111dafa-kube-api-access-mmzzm\") pod \"network-metrics-daemon-xfgjq\" (UID: \"bc719829-66b1-4a84-b994-e71cf111dafa\") " pod="openshift-multus/network-metrics-daemon-xfgjq"
Sep 16 17:24:50.027114 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:50.027091    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a1223407-0c9e-4ed6-bd17-cf0f3962e27e-multus-conf-dir\") pod \"multus-59rw5\" (UID: \"a1223407-0c9e-4ed6-bd17-cf0f3962e27e\") " pod="openshift-multus/multus-59rw5"
Sep 16 17:24:50.027114 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:50.027100    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x46mt\" (UniqueName: \"kubernetes.io/projected/1d620b9d-0055-4ed6-aced-2f4cfc54cbd8-kube-api-access-x46mt\") pod \"network-check-target-4c79x\" (UID: \"1d620b9d-0055-4ed6-aced-2f4cfc54cbd8\") " pod="openshift-network-diagnostics/network-check-target-4c79x"
Sep 16 17:24:50.027372 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:50.027143    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c2617cf7-6369-4fc4-9fb5-6e6cfc4e9add-run-openvswitch\") pod \"ovnkube-node-f9qxl\" (UID: \"c2617cf7-6369-4fc4-9fb5-6e6cfc4e9add\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9qxl"
Sep 16 17:24:50.027372 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:50.027181    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c2617cf7-6369-4fc4-9fb5-6e6cfc4e9add-ovnkube-script-lib\") pod \"ovnkube-node-f9qxl\" (UID: \"c2617cf7-6369-4fc4-9fb5-6e6cfc4e9add\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9qxl"
Sep 16 17:24:50.027372 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:50.027219    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5df1fb59-c460-463c-8691-2c270b69aa34-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6fsn7\" (UID: \"5df1fb59-c460-463c-8691-2c270b69aa34\") " pod="openshift-multus/multus-additional-cni-plugins-6fsn7"
Sep 16 17:24:50.027372 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:50.027248    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6bfc5b33-9432-43a1-a89a-8a2f18acc45f-tmp\") pod \"tuned-pgq7f\" (UID: \"6bfc5b33-9432-43a1-a89a-8a2f18acc45f\") " pod="openshift-cluster-node-tuning-operator/tuned-pgq7f"
Sep 16 17:24:50.027372 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:50.027278    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a1223407-0c9e-4ed6-bd17-cf0f3962e27e-multus-socket-dir-parent\") pod \"multus-59rw5\" (UID: \"a1223407-0c9e-4ed6-bd17-cf0f3962e27e\") " pod="openshift-multus/multus-59rw5"
Sep 16 17:24:50.027372 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:50.027310    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6bfc5b33-9432-43a1-a89a-8a2f18acc45f-var-lib-kubelet\") pod \"tuned-pgq7f\" (UID: \"6bfc5b33-9432-43a1-a89a-8a2f18acc45f\") " pod="openshift-cluster-node-tuning-operator/tuned-pgq7f"
Sep 16 17:24:50.027629 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:50.027378    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/1ea5d21d-1046-4edb-b1d9-9ca43ebc564d-iptables-alerter-script\") pod \"iptables-alerter-gmmwp\" (UID: \"1ea5d21d-1046-4edb-b1d9-9ca43ebc564d\") " pod="openshift-network-operator/iptables-alerter-gmmwp"
Sep 16 17:24:50.027629 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:50.027436    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkjgp\" (UniqueName: \"kubernetes.io/projected/1ea5d21d-1046-4edb-b1d9-9ca43ebc564d-kube-api-access-vkjgp\") pod \"iptables-alerter-gmmwp\" (UID: \"1ea5d21d-1046-4edb-b1d9-9ca43ebc564d\") " pod="openshift-network-operator/iptables-alerter-gmmwp"
Sep 16 17:24:50.027629 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:50.027447    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a1223407-0c9e-4ed6-bd17-cf0f3962e27e-multus-socket-dir-parent\") pod \"multus-59rw5\" (UID: \"a1223407-0c9e-4ed6-bd17-cf0f3962e27e\") " pod="openshift-multus/multus-59rw5"
Sep 16 17:24:50.027629 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:50.027462    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a1223407-0c9e-4ed6-bd17-cf0f3962e27e-etc-kubernetes\") pod \"multus-59rw5\" (UID: \"a1223407-0c9e-4ed6-bd17-cf0f3962e27e\") " pod="openshift-multus/multus-59rw5"
Sep 16 17:24:50.027629 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:50.027494    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c2617cf7-6369-4fc4-9fb5-6e6cfc4e9add-ovnkube-config\") pod \"ovnkube-node-f9qxl\" (UID: \"c2617cf7-6369-4fc4-9fb5-6e6cfc4e9add\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9qxl"
Sep 16 17:24:50.027629 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:50.027506    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a1223407-0c9e-4ed6-bd17-cf0f3962e27e-etc-kubernetes\") pod \"multus-59rw5\" (UID: \"a1223407-0c9e-4ed6-bd17-cf0f3962e27e\") " pod="openshift-multus/multus-59rw5"
Sep 16 17:24:50.027629 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:50.027518    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5df1fb59-c460-463c-8691-2c270b69aa34-cnibin\") pod \"multus-additional-cni-plugins-6fsn7\" (UID: \"5df1fb59-c460-463c-8691-2c270b69aa34\") " pod="openshift-multus/multus-additional-cni-plugins-6fsn7"
Sep 16 17:24:50.027953 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:50.027729    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6bfc5b33-9432-43a1-a89a-8a2f18acc45f-host\") pod \"tuned-pgq7f\" (UID: \"6bfc5b33-9432-43a1-a89a-8a2f18acc45f\") " pod="openshift-cluster-node-tuning-operator/tuned-pgq7f"
Sep 16 17:24:50.027953 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:50.027767    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/21a2db47-94e4-4d2b-9cb2-0b2bca64ef5b-hosts-file\") pod \"node-resolver-qtdvz\" (UID: \"21a2db47-94e4-4d2b-9cb2-0b2bca64ef5b\") " pod="openshift-dns/node-resolver-qtdvz"
Sep 16 17:24:50.027953 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:50.027782    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/6bfc5b33-9432-43a1-a89a-8a2f18acc45f-etc-sysconfig\") pod \"tuned-pgq7f\" (UID: \"6bfc5b33-9432-43a1-a89a-8a2f18acc45f\") " pod="openshift-cluster-node-tuning-operator/tuned-pgq7f"
Sep 16 17:24:50.027953 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:50.027816    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/41b48733-fd0e-4eaa-a91c-5d57f642b2f8-host\") pod \"node-ca-t6cx6\" (UID: \"41b48733-fd0e-4eaa-a91c-5d57f642b2f8\") " pod="openshift-image-registry/node-ca-t6cx6"
Sep 16 17:24:50.027953 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:50.027863    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a1223407-0c9e-4ed6-bd17-cf0f3962e27e-cni-binary-copy\") pod \"multus-59rw5\" (UID: \"a1223407-0c9e-4ed6-bd17-cf0f3962e27e\") " pod="openshift-multus/multus-59rw5"
Sep 16 17:24:50.027953 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:50.027879    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c2617cf7-6369-4fc4-9fb5-6e6cfc4e9add-host-kubelet\") pod \"ovnkube-node-f9qxl\" (UID: \"c2617cf7-6369-4fc4-9fb5-6e6cfc4e9add\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9qxl"
Sep 16 17:24:50.027953 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:50.027897    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c73a8482-2abb-407e-972d-05a54828be7a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-w58mp\" (UID: \"c73a8482-2abb-407e-972d-05a54828be7a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w58mp"
Sep 16 17:24:50.027953 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:50.027947    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a1223407-0c9e-4ed6-bd17-cf0f3962e27e-host-var-lib-cni-multus\") pod \"multus-59rw5\" (UID: \"a1223407-0c9e-4ed6-bd17-cf0f3962e27e\") " pod="openshift-multus/multus-59rw5"
Sep 16 17:24:50.028323 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:50.027975    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c2617cf7-6369-4fc4-9fb5-6e6cfc4e9add-run-ovn\") pod \"ovnkube-node-f9qxl\" (UID: \"c2617cf7-6369-4fc4-9fb5-6e6cfc4e9add\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9qxl"
Sep 16 17:24:50.028323 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:50.027996    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c2617cf7-6369-4fc4-9fb5-6e6cfc4e9add-host-cni-netd\") pod \"ovnkube-node-f9qxl\" (UID: \"c2617cf7-6369-4fc4-9fb5-6e6cfc4e9add\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9qxl"
Sep 16 17:24:50.028323 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:50.028017    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/6bfc5b33-9432-43a1-a89a-8a2f18acc45f-etc-modprobe-d\") pod \"tuned-pgq7f\" (UID: \"6bfc5b33-9432-43a1-a89a-8a2f18acc45f\") " pod="openshift-cluster-node-tuning-operator/tuned-pgq7f"
Sep 16 17:24:50.028323 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:50.028037    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/6bfc5b33-9432-43a1-a89a-8a2f18acc45f-etc-sysctl-d\") pod \"tuned-pgq7f\" (UID: \"6bfc5b33-9432-43a1-a89a-8a2f18acc45f\") " pod="openshift-cluster-node-tuning-operator/tuned-pgq7f"
Sep 16 17:24:50.028323 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:50.028062    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ef27aa30-cb45-45bc-8d0e-50107fc6459d-agent-certs\") pod \"konnectivity-agent-gntnz\" (UID: \"ef27aa30-cb45-45bc-8d0e-50107fc6459d\") " pod="kube-system/konnectivity-agent-gntnz"
Sep 16 17:24:50.028323 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:50.028062    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c73a8482-2abb-407e-972d-05a54828be7a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-w58mp\" (UID: \"c73a8482-2abb-407e-972d-05a54828be7a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w58mp"
Sep 16 17:24:50.028323 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:50.028099    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a1223407-0c9e-4ed6-bd17-cf0f3962e27e-system-cni-dir\") pod \"multus-59rw5\" (UID: \"a1223407-0c9e-4ed6-bd17-cf0f3962e27e\") " pod="openshift-multus/multus-59rw5"
Sep 16 17:24:50.028323 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:50.028110    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a1223407-0c9e-4ed6-bd17-cf0f3962e27e-host-var-lib-cni-multus\") pod \"multus-59rw5\" (UID: \"a1223407-0c9e-4ed6-bd17-cf0f3962e27e\") " pod="openshift-multus/multus-59rw5"
Sep 16 17:24:50.028323 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:50.028192    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/1ea5d21d-1046-4edb-b1d9-9ca43ebc564d-iptables-alerter-script\") pod \"iptables-alerter-gmmwp\" (UID: \"1ea5d21d-1046-4edb-b1d9-9ca43ebc564d\") " pod="openshift-network-operator/iptables-alerter-gmmwp"
Sep 16 17:24:50.028323 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:50.028205    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a1223407-0c9e-4ed6-bd17-cf0f3962e27e-system-cni-dir\") pod \"multus-59rw5\" (UID: \"a1223407-0c9e-4ed6-bd17-cf0f3962e27e\") " pod="openshift-multus/multus-59rw5"
Sep 16 17:24:50.028323 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:50.028269    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c2617cf7-6369-4fc4-9fb5-6e6cfc4e9add-host-cni-bin\") pod \"ovnkube-node-f9qxl\" (UID: \"c2617cf7-6369-4fc4-9fb5-6e6cfc4e9add\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9qxl"
Sep 16 17:24:50.028323 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:50.028294    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5df1fb59-c460-463c-8691-2c270b69aa34-system-cni-dir\") pod \"multus-additional-cni-plugins-6fsn7\" (UID: \"5df1fb59-c460-463c-8691-2c270b69aa34\") " pod="openshift-multus/multus-additional-cni-plugins-6fsn7"
Sep 16 17:24:50.028871 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:50.028337    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6bfc5b33-9432-43a1-a89a-8a2f18acc45f-etc-kubernetes\") pod \"tuned-pgq7f\" (UID: \"6bfc5b33-9432-43a1-a89a-8a2f18acc45f\") " pod="openshift-cluster-node-tuning-operator/tuned-pgq7f"
Sep 16 17:24:50.028871 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:50.028380    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/6bfc5b33-9432-43a1-a89a-8a2f18acc45f-run\") pod \"tuned-pgq7f\" (UID: \"6bfc5b33-9432-43a1-a89a-8a2f18acc45f\") " pod="openshift-cluster-node-tuning-operator/tuned-pgq7f"
Sep 16 17:24:50.028871 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:50.028404    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/41b48733-fd0e-4eaa-a91c-5d57f642b2f8-serviceca\") pod \"node-ca-t6cx6\" (UID: \"41b48733-fd0e-4eaa-a91c-5d57f642b2f8\") " pod="openshift-image-registry/node-ca-t6cx6"
Sep 16 17:24:50.028871 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:50.028429    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a1223407-0c9e-4ed6-bd17-cf0f3962e27e-host-var-lib-kubelet\") pod \"multus-59rw5\" (UID: \"a1223407-0c9e-4ed6-bd17-cf0f3962e27e\") " pod="openshift-multus/multus-59rw5"
Sep 16 17:24:50.028871 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:50.028454    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a1223407-0c9e-4ed6-bd17-cf0f3962e27e-multus-daemon-config\") pod \"multus-59rw5\" (UID: \"a1223407-0c9e-4ed6-bd17-cf0f3962e27e\") " pod="openshift-multus/multus-59rw5"
Sep 16 17:24:50.028871 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:50.028470    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a1223407-0c9e-4ed6-bd17-cf0f3962e27e-cni-binary-copy\") pod \"multus-59rw5\" (UID: \"a1223407-0c9e-4ed6-bd17-cf0f3962e27e\") " pod="openshift-multus/multus-59rw5"
Sep 16 17:24:50.028871 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:50.028483    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a1223407-0c9e-4ed6-bd17-cf0f3962e27e-host-var-lib-kubelet\") pod \"multus-59rw5\" (UID: \"a1223407-0c9e-4ed6-bd17-cf0f3962e27e\") " pod="openshift-multus/multus-59rw5"
Sep 16 17:24:50.028871 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:50.028476    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c2617cf7-6369-4fc4-9fb5-6e6cfc4e9add-ovn-node-metrics-cert\") pod \"ovnkube-node-f9qxl\" (UID: \"c2617cf7-6369-4fc4-9fb5-6e6cfc4e9add\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9qxl"
Sep 16 17:24:50.028871 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:50.028518    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6l4d\" (UniqueName: \"kubernetes.io/projected/c2617cf7-6369-4fc4-9fb5-6e6cfc4e9add-kube-api-access-n6l4d\") pod \"ovnkube-node-f9qxl\" (UID: \"c2617cf7-6369-4fc4-9fb5-6e6cfc4e9add\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9qxl"
Sep 16 17:24:50.028871 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:50.028550    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5df1fb59-c460-463c-8691-2c270b69aa34-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6fsn7\" (UID: \"5df1fb59-c460-463c-8691-2c270b69aa34\") " pod="openshift-multus/multus-additional-cni-plugins-6fsn7"
Sep 16 17:24:50.028871 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:50.028568    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/6bfc5b33-9432-43a1-a89a-8a2f18acc45f-etc-sysctl-conf\") pod \"tuned-pgq7f\" (UID: \"6bfc5b33-9432-43a1-a89a-8a2f18acc45f\") " pod="openshift-cluster-node-tuning-operator/tuned-pgq7f"
Sep 16 17:24:50.028871 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:50.028585    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6bfc5b33-9432-43a1-a89a-8a2f18acc45f-sys\") pod \"tuned-pgq7f\" (UID: \"6bfc5b33-9432-43a1-a89a-8a2f18acc45f\") " pod="openshift-cluster-node-tuning-operator/tuned-pgq7f"
Sep 16 17:24:50.028871 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:50.028610    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c73a8482-2abb-407e-972d-05a54828be7a-socket-dir\") pod \"aws-ebs-csi-driver-node-w58mp\" (UID: \"c73a8482-2abb-407e-972d-05a54828be7a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w58mp"
Sep 16 17:24:50.028871 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:50.028634    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a1223407-0c9e-4ed6-bd17-cf0f3962e27e-host-var-lib-cni-bin\") pod \"multus-59rw5\" (UID: \"a1223407-0c9e-4ed6-bd17-cf0f3962e27e\") " pod="openshift-multus/multus-59rw5"
Sep 16 17:24:50.028871 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:50.028656    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a1223407-0c9e-4ed6-bd17-cf0f3962e27e-hostroot\") pod \"multus-59rw5\" (UID: \"a1223407-0c9e-4ed6-bd17-cf0f3962e27e\") " pod="openshift-multus/multus-59rw5"
Sep 16 17:24:50.028871 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:50.028680    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21a2db47-94e4-4d2b-9cb2-0b2bca64ef5b-tmp-dir\") pod \"node-resolver-qtdvz\" (UID: \"21a2db47-94e4-4d2b-9cb2-0b2bca64ef5b\") " pod="openshift-dns/node-resolver-qtdvz"
Sep 16 17:24:50.028871 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:50.028726    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e9240885-14d4-4b43-bf98-442e30d61c36-original-pull-secret\") pod \"global-pull-secret-syncer-s7zg6\" (UID: \"e9240885-14d4-4b43-bf98-442e30d61c36\") " pod="kube-system/global-pull-secret-syncer-s7zg6"
Sep 16 17:24:50.029601 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:50.028728    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a1223407-0c9e-4ed6-bd17-cf0f3962e27e-hostroot\") pod \"multus-59rw5\" (UID: \"a1223407-0c9e-4ed6-bd17-cf0f3962e27e\") " pod="openshift-multus/multus-59rw5"
Sep 16 17:24:50.029601 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:50.028744    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c73a8482-2abb-407e-972d-05a54828be7a-socket-dir\") pod \"aws-ebs-csi-driver-node-w58mp\" (UID: \"c73a8482-2abb-407e-972d-05a54828be7a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w58mp"
Sep 16 17:24:50.029601 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:50.028740    2579 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 16 17:24:50.029601 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:50.028775    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a1223407-0c9e-4ed6-bd17-cf0f3962e27e-host-var-lib-cni-bin\") pod \"multus-59rw5\" (UID: \"a1223407-0c9e-4ed6-bd17-cf0f3962e27e\") " pod="openshift-multus/multus-59rw5"
Sep 16 17:24:50.029601 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:50.028759    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c2617cf7-6369-4fc4-9fb5-6e6cfc4e9add-host-run-netns\") pod \"ovnkube-node-f9qxl\" (UID: \"c2617cf7-6369-4fc4-9fb5-6e6cfc4e9add\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9qxl"
Sep 16 17:24:50.029601 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:50.028863    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c2617cf7-6369-4fc4-9fb5-6e6cfc4e9add-log-socket\") pod \"ovnkube-node-f9qxl\" (UID: \"c2617cf7-6369-4fc4-9fb5-6e6cfc4e9add\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9qxl"
Sep 16 17:24:50.029601 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:50.028889    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hskq\" (UniqueName: \"kubernetes.io/projected/6bfc5b33-9432-43a1-a89a-8a2f18acc45f-kube-api-access-8hskq\") pod \"tuned-pgq7f\" (UID: \"6bfc5b33-9432-43a1-a89a-8a2f18acc45f\") " pod="openshift-cluster-node-tuning-operator/tuned-pgq7f"
Sep 16 17:24:50.029601 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:50.028912    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzg2d\" (UniqueName: \"kubernetes.io/projected/41b48733-fd0e-4eaa-a91c-5d57f642b2f8-kube-api-access-wzg2d\") pod \"node-ca-t6cx6\" (UID: \"41b48733-fd0e-4eaa-a91c-5d57f642b2f8\") " pod="openshift-image-registry/node-ca-t6cx6"
Sep 16 17:24:50.029601 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:50.028913    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a1223407-0c9e-4ed6-bd17-cf0f3962e27e-multus-daemon-config\") pod \"multus-59rw5\" (UID: \"a1223407-0c9e-4ed6-bd17-cf0f3962e27e\") " pod="openshift-multus/multus-59rw5"
Sep 16 17:24:50.029601 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:50.028937    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c2617cf7-6369-4fc4-9fb5-6e6cfc4e9add-host-run-ovn-kubernetes\") pod \"ovnkube-node-f9qxl\" (UID: \"c2617cf7-6369-4fc4-9fb5-6e6cfc4e9add\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9qxl"
Sep 16 17:24:50.029601 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:50.028987    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c2617cf7-6369-4fc4-9fb5-6e6cfc4e9add-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-f9qxl\" (UID: \"c2617cf7-6369-4fc4-9fb5-6e6cfc4e9add\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9qxl"
Sep 16 17:24:50.029601 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:50.029017    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxqrk\" (UniqueName: \"kubernetes.io/projected/21a2db47-94e4-4d2b-9cb2-0b2bca64ef5b-kube-api-access-mxqrk\") pod \"node-resolver-qtdvz\" (UID: \"21a2db47-94e4-4d2b-9cb2-0b2bca64ef5b\") " pod="openshift-dns/node-resolver-qtdvz"
Sep 16 17:24:50.029601 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:50.029062    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bc719829-66b1-4a84-b994-e71cf111dafa-metrics-certs\") pod \"network-metrics-daemon-xfgjq\" (UID: \"bc719829-66b1-4a84-b994-e71cf111dafa\") " pod="openshift-multus/network-metrics-daemon-xfgjq"
Sep 16 17:24:50.029601 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:50.029086    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5df1fb59-c460-463c-8691-2c270b69aa34-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6fsn7\" (UID: \"5df1fb59-c460-463c-8691-2c270b69aa34\") " pod="openshift-multus/multus-additional-cni-plugins-6fsn7"
Sep 16 17:24:50.029601 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:50.029121    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6bfc5b33-9432-43a1-a89a-8a2f18acc45f-lib-modules\") pod \"tuned-pgq7f\" (UID: \"6bfc5b33-9432-43a1-a89a-8a2f18acc45f\") " pod="openshift-cluster-node-tuning-operator/tuned-pgq7f"
Sep 16 17:24:50.029601 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:50.029152    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a1223407-0c9e-4ed6-bd17-cf0f3962e27e-cnibin\") pod \"multus-59rw5\" (UID: \"a1223407-0c9e-4ed6-bd17-cf0f3962e27e\") " pod="openshift-multus/multus-59rw5"
Sep 16 17:24:50.029601 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:50.029192    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a1223407-0c9e-4ed6-bd17-cf0f3962e27e-host-run-k8s-cni-cncf-io\") pod \"multus-59rw5\" (UID: \"a1223407-0c9e-4ed6-bd17-cf0f3962e27e\") " pod="openshift-multus/multus-59rw5"
Sep 16 17:24:50.030370 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:50.029216    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a1223407-0c9e-4ed6-bd17-cf0f3962e27e-cnibin\") pod \"multus-59rw5\" (UID: \"a1223407-0c9e-4ed6-bd17-cf0f3962e27e\") " pod="openshift-multus/multus-59rw5"
Sep 16 17:24:50.030370 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:50.029250    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a1223407-0c9e-4ed6-bd17-cf0f3962e27e-host-run-k8s-cni-cncf-io\") pod \"multus-59rw5\" (UID: \"a1223407-0c9e-4ed6-bd17-cf0f3962e27e\") " pod="openshift-multus/multus-59rw5"
Sep 16 17:24:50.030370 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:50.029253    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a1223407-0c9e-4ed6-bd17-cf0f3962e27e-host-run-multus-certs\") pod \"multus-59rw5\" (UID: \"a1223407-0c9e-4ed6-bd17-cf0f3962e27e\") " pod="openshift-multus/multus-59rw5"
Sep 16 17:24:50.030370 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:50.029285    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c2617cf7-6369-4fc4-9fb5-6e6cfc4e9add-var-lib-openvswitch\") pod \"ovnkube-node-f9qxl\" (UID: \"c2617cf7-6369-4fc4-9fb5-6e6cfc4e9add\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9qxl"
Sep 16 17:24:50.030370 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:50.029288    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a1223407-0c9e-4ed6-bd17-cf0f3962e27e-host-run-multus-certs\") pod \"multus-59rw5\" (UID: \"a1223407-0c9e-4ed6-bd17-cf0f3962e27e\") " pod="openshift-multus/multus-59rw5"
Sep 16 17:24:50.030370 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:50.029311    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c2617cf7-6369-4fc4-9fb5-6e6cfc4e9add-etc-openvswitch\") pod \"ovnkube-node-f9qxl\" (UID: \"c2617cf7-6369-4fc4-9fb5-6e6cfc4e9add\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9qxl"
Sep 16 17:24:50.030370 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:50.029333    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e9240885-14d4-4b43-bf98-442e30d61c36-dbus\") pod \"global-pull-secret-syncer-s7zg6\" (UID: \"e9240885-14d4-4b43-bf98-442e30d61c36\") " pod="kube-system/global-pull-secret-syncer-s7zg6"
Sep 16 17:24:50.030370 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:50.029355    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/6bfc5b33-9432-43a1-a89a-8a2f18acc45f-etc-tuned\") pod \"tuned-pgq7f\" (UID: \"6bfc5b33-9432-43a1-a89a-8a2f18acc45f\") " pod="openshift-cluster-node-tuning-operator/tuned-pgq7f"
Sep 16 17:24:50.030370 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:50.029378    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1ea5d21d-1046-4edb-b1d9-9ca43ebc564d-host-slash\") pod \"iptables-alerter-gmmwp\" (UID: \"1ea5d21d-1046-4edb-b1d9-9ca43ebc564d\") " pod="openshift-network-operator/iptables-alerter-gmmwp"
Sep 16 17:24:50.030370 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:50.029405    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a1223407-0c9e-4ed6-bd17-cf0f3962e27e-host-run-netns\") pod \"multus-59rw5\" (UID: \"a1223407-0c9e-4ed6-bd17-cf0f3962e27e\") " pod="openshift-multus/multus-59rw5"
Sep 16 17:24:50.030370 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:50.029412    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1ea5d21d-1046-4edb-b1d9-9ca43ebc564d-host-slash\") pod \"iptables-alerter-gmmwp\" (UID: \"1ea5d21d-1046-4edb-b1d9-9ca43ebc564d\") " pod="openshift-network-operator/iptables-alerter-gmmwp"
Sep 16 17:24:50.030370 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:50.029427    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvjm2\" (UniqueName: \"kubernetes.io/projected/a1223407-0c9e-4ed6-bd17-cf0f3962e27e-kube-api-access-nvjm2\") pod \"multus-59rw5\" (UID: \"a1223407-0c9e-4ed6-bd17-cf0f3962e27e\") " pod="openshift-multus/multus-59rw5"
Sep 16 17:24:50.030370 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:50.029456    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a1223407-0c9e-4ed6-bd17-cf0f3962e27e-host-run-netns\") pod \"multus-59rw5\" (UID: \"a1223407-0c9e-4ed6-bd17-cf0f3962e27e\") " pod="openshift-multus/multus-59rw5"
Sep 16 17:24:50.030370 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:50.029489    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c2617cf7-6369-4fc4-9fb5-6e6cfc4e9add-run-systemd\") pod \"ovnkube-node-f9qxl\" (UID: \"c2617cf7-6369-4fc4-9fb5-6e6cfc4e9add\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9qxl"
Sep 16 17:24:50.030370 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:50.029514    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e9240885-14d4-4b43-bf98-442e30d61c36-kubelet-config\") pod \"global-pull-secret-syncer-s7zg6\" (UID: \"e9240885-14d4-4b43-bf98-442e30d61c36\") " pod="kube-system/global-pull-secret-syncer-s7zg6"
Sep 16 17:24:50.030370 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:50.029536    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5df1fb59-c460-463c-8691-2c270b69aa34-cni-binary-copy\") pod \"multus-additional-cni-plugins-6fsn7\" (UID: \"5df1fb59-c460-463c-8691-2c270b69aa34\") " pod="openshift-multus/multus-additional-cni-plugins-6fsn7"
Sep 16 17:24:50.030370 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:50.029562    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c73a8482-2abb-407e-972d-05a54828be7a-device-dir\") pod \"aws-ebs-csi-driver-node-w58mp\" (UID: \"c73a8482-2abb-407e-972d-05a54828be7a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w58mp"
Sep 16 17:24:50.031098 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:50.029602    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a1223407-0c9e-4ed6-bd17-cf0f3962e27e-multus-cni-dir\") pod \"multus-59rw5\" (UID: \"a1223407-0c9e-4ed6-bd17-cf0f3962e27e\") " pod="openshift-multus/multus-59rw5"
Sep 16 17:24:50.031098 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:50.029641    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c2617cf7-6369-4fc4-9fb5-6e6cfc4e9add-host-slash\") pod \"ovnkube-node-f9qxl\" (UID: \"c2617cf7-6369-4fc4-9fb5-6e6cfc4e9add\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9qxl"
Sep 16 17:24:50.031098 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:50.029651    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c73a8482-2abb-407e-972d-05a54828be7a-device-dir\") pod \"aws-ebs-csi-driver-node-w58mp\" (UID: \"c73a8482-2abb-407e-972d-05a54828be7a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w58mp"
Sep 16 17:24:50.031098 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:50.029668    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/6bfc5b33-9432-43a1-a89a-8a2f18acc45f-etc-systemd\") pod \"tuned-pgq7f\" (UID: \"6bfc5b33-9432-43a1-a89a-8a2f18acc45f\") " pod="openshift-cluster-node-tuning-operator/tuned-pgq7f"
Sep 16 17:24:50.031098 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:50.029695    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c73a8482-2abb-407e-972d-05a54828be7a-registration-dir\") pod \"aws-ebs-csi-driver-node-w58mp\" (UID: \"c73a8482-2abb-407e-972d-05a54828be7a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w58mp"
Sep 16 17:24:50.031098 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:50.029728    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c73a8482-2abb-407e-972d-05a54828be7a-etc-selinux\") pod \"aws-ebs-csi-driver-node-w58mp\" (UID: \"c73a8482-2abb-407e-972d-05a54828be7a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w58mp"
Sep 16 17:24:50.031098 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:50.029754    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c73a8482-2abb-407e-972d-05a54828be7a-sys-fs\") pod \"aws-ebs-csi-driver-node-w58mp\" (UID: \"c73a8482-2abb-407e-972d-05a54828be7a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w58mp"
Sep 16 17:24:50.031098 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:50.029788    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chp67\" (UniqueName: \"kubernetes.io/projected/5df1fb59-c460-463c-8691-2c270b69aa34-kube-api-access-chp67\") pod \"multus-additional-cni-plugins-6fsn7\" (UID: \"5df1fb59-c460-463c-8691-2c270b69aa34\") " pod="openshift-multus/multus-additional-cni-plugins-6fsn7"
Sep 16 17:24:50.031098 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:50.029819    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vbr5\" (UniqueName: \"kubernetes.io/projected/c73a8482-2abb-407e-972d-05a54828be7a-kube-api-access-6vbr5\") pod \"aws-ebs-csi-driver-node-w58mp\" (UID: \"c73a8482-2abb-407e-972d-05a54828be7a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w58mp"
Sep 16 17:24:50.031098 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:50.029828    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a1223407-0c9e-4ed6-bd17-cf0f3962e27e-multus-cni-dir\") pod \"multus-59rw5\" (UID: \"a1223407-0c9e-4ed6-bd17-cf0f3962e27e\") " pod="openshift-multus/multus-59rw5"
Sep 16 17:24:50.031098 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:50.029836    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c73a8482-2abb-407e-972d-05a54828be7a-etc-selinux\") pod \"aws-ebs-csi-driver-node-w58mp\" (UID: \"c73a8482-2abb-407e-972d-05a54828be7a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w58mp"
Sep 16 17:24:50.031098 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:50.029884    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ef27aa30-cb45-45bc-8d0e-50107fc6459d-konnectivity-ca\") pod \"konnectivity-agent-gntnz\" (UID: \"ef27aa30-cb45-45bc-8d0e-50107fc6459d\") " pod="kube-system/konnectivity-agent-gntnz"
Sep 16 17:24:50.031098 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:50.029890    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c73a8482-2abb-407e-972d-05a54828be7a-sys-fs\") pod \"aws-ebs-csi-driver-node-w58mp\" (UID: \"c73a8482-2abb-407e-972d-05a54828be7a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w58mp"
Sep 16 17:24:50.031098 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:50.029982    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c73a8482-2abb-407e-972d-05a54828be7a-registration-dir\") pod \"aws-ebs-csi-driver-node-w58mp\" (UID: \"c73a8482-2abb-407e-972d-05a54828be7a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w58mp"
Sep 16 17:24:50.031098 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:50.030428    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ef27aa30-cb45-45bc-8d0e-50107fc6459d-konnectivity-ca\") pod \"konnectivity-agent-gntnz\" (UID: \"ef27aa30-cb45-45bc-8d0e-50107fc6459d\") " pod="kube-system/konnectivity-agent-gntnz"
Sep 16 17:24:50.034089 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:50.034066    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ef27aa30-cb45-45bc-8d0e-50107fc6459d-agent-certs\") pod \"konnectivity-agent-gntnz\" (UID: \"ef27aa30-cb45-45bc-8d0e-50107fc6459d\") " pod="kube-system/konnectivity-agent-gntnz"
Sep 16 17:24:50.040402 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:50.040195    2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-4xl76"
Sep 16 17:24:50.044617 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:50.044596    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkjgp\" (UniqueName: \"kubernetes.io/projected/1ea5d21d-1046-4edb-b1d9-9ca43ebc564d-kube-api-access-vkjgp\") pod \"iptables-alerter-gmmwp\" (UID: \"1ea5d21d-1046-4edb-b1d9-9ca43ebc564d\") " pod="openshift-network-operator/iptables-alerter-gmmwp"
Sep 16 17:24:50.044737 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:50.044596    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvjm2\" (UniqueName: \"kubernetes.io/projected/a1223407-0c9e-4ed6-bd17-cf0f3962e27e-kube-api-access-nvjm2\") pod \"multus-59rw5\" (UID: \"a1223407-0c9e-4ed6-bd17-cf0f3962e27e\") " pod="openshift-multus/multus-59rw5"
Sep 16 17:24:50.055139 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:50.055073    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vbr5\" (UniqueName: \"kubernetes.io/projected/c73a8482-2abb-407e-972d-05a54828be7a-kube-api-access-6vbr5\") pod \"aws-ebs-csi-driver-node-w58mp\" (UID: \"c73a8482-2abb-407e-972d-05a54828be7a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w58mp"
Sep 16 17:24:50.130526 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:50.130486    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c2617cf7-6369-4fc4-9fb5-6e6cfc4e9add-ovnkube-config\") pod \"ovnkube-node-f9qxl\" (UID: \"c2617cf7-6369-4fc4-9fb5-6e6cfc4e9add\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9qxl"
Sep 16 17:24:50.130681 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:50.130533    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5df1fb59-c460-463c-8691-2c270b69aa34-cnibin\") pod \"multus-additional-cni-plugins-6fsn7\" (UID: \"5df1fb59-c460-463c-8691-2c270b69aa34\") " pod="openshift-multus/multus-additional-cni-plugins-6fsn7"
Sep 16 17:24:50.130681 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:50.130555    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6bfc5b33-9432-43a1-a89a-8a2f18acc45f-host\") pod \"tuned-pgq7f\" (UID: \"6bfc5b33-9432-43a1-a89a-8a2f18acc45f\") " pod="openshift-cluster-node-tuning-operator/tuned-pgq7f"
Sep 16 17:24:50.130681 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:50.130578    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/21a2db47-94e4-4d2b-9cb2-0b2bca64ef5b-hosts-file\") pod \"node-resolver-qtdvz\" (UID: \"21a2db47-94e4-4d2b-9cb2-0b2bca64ef5b\") " pod="openshift-dns/node-resolver-qtdvz"
Sep 16 17:24:50.130681 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:50.130600    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/6bfc5b33-9432-43a1-a89a-8a2f18acc45f-etc-sysconfig\") pod \"tuned-pgq7f\" (UID: \"6bfc5b33-9432-43a1-a89a-8a2f18acc45f\") " pod="openshift-cluster-node-tuning-operator/tuned-pgq7f"
Sep 16 17:24:50.130681 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:50.130620    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/41b48733-fd0e-4eaa-a91c-5d57f642b2f8-host\") pod \"node-ca-t6cx6\" (UID: \"41b48733-fd0e-4eaa-a91c-5d57f642b2f8\") " pod="openshift-image-registry/node-ca-t6cx6"
Sep 16 17:24:50.130681 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:50.130641    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c2617cf7-6369-4fc4-9fb5-6e6cfc4e9add-host-kubelet\") pod \"ovnkube-node-f9qxl\" (UID: \"c2617cf7-6369-4fc4-9fb5-6e6cfc4e9add\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9qxl"
Sep 16 17:24:50.130681 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:50.130640    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5df1fb59-c460-463c-8691-2c270b69aa34-cnibin\") pod \"multus-additional-cni-plugins-6fsn7\" (UID: \"5df1fb59-c460-463c-8691-2c270b69aa34\") " pod="openshift-multus/multus-additional-cni-plugins-6fsn7"
Sep 16 17:24:50.130681 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:50.130664    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c2617cf7-6369-4fc4-9fb5-6e6cfc4e9add-run-ovn\") pod \"ovnkube-node-f9qxl\" (UID: \"c2617cf7-6369-4fc4-9fb5-6e6cfc4e9add\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9qxl"
Sep 16 17:24:50.131095 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:50.130686    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c2617cf7-6369-4fc4-9fb5-6e6cfc4e9add-host-cni-netd\") pod \"ovnkube-node-f9qxl\" (UID: \"c2617cf7-6369-4fc4-9fb5-6e6cfc4e9add\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9qxl"
Sep 16 17:24:50.131095 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:50.130690    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/6bfc5b33-9432-43a1-a89a-8a2f18acc45f-etc-sysconfig\") pod \"tuned-pgq7f\" (UID: \"6bfc5b33-9432-43a1-a89a-8a2f18acc45f\") " pod="openshift-cluster-node-tuning-operator/tuned-pgq7f"
Sep 16 17:24:50.131095 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:50.130707    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/6bfc5b33-9432-43a1-a89a-8a2f18acc45f-etc-modprobe-d\") pod \"tuned-pgq7f\" (UID: \"6bfc5b33-9432-43a1-a89a-8a2f18acc45f\") " pod="openshift-cluster-node-tuning-operator/tuned-pgq7f"
Sep 16 17:24:50.131095 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:50.130711    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/21a2db47-94e4-4d2b-9cb2-0b2bca64ef5b-hosts-file\") pod \"node-resolver-qtdvz\" (UID: \"21a2db47-94e4-4d2b-9cb2-0b2bca64ef5b\") " pod="openshift-dns/node-resolver-qtdvz"
Sep 16 17:24:50.131095 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:50.130690    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6bfc5b33-9432-43a1-a89a-8a2f18acc45f-host\") pod \"tuned-pgq7f\" (UID: \"6bfc5b33-9432-43a1-a89a-8a2f18acc45f\") " pod="openshift-cluster-node-tuning-operator/tuned-pgq7f"
Sep 16 17:24:50.131095 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:50.130729    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/6bfc5b33-9432-43a1-a89a-8a2f18acc45f-etc-sysctl-d\") pod \"tuned-pgq7f\" (UID: \"6bfc5b33-9432-43a1-a89a-8a2f18acc45f\") " pod="openshift-cluster-node-tuning-operator/tuned-pgq7f"
Sep 16 17:24:50.131095 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:50.130748    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/41b48733-fd0e-4eaa-a91c-5d57f642b2f8-host\") pod \"node-ca-t6cx6\" (UID: \"41b48733-fd0e-4eaa-a91c-5d57f642b2f8\") " pod="openshift-image-registry/node-ca-t6cx6"
Sep 16 17:24:50.131095 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:50.130754    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c2617cf7-6369-4fc4-9fb5-6e6cfc4e9add-host-cni-bin\") pod \"ovnkube-node-f9qxl\" (UID: \"c2617cf7-6369-4fc4-9fb5-6e6cfc4e9add\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9qxl"
Sep 16 17:24:50.131095 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:50.130760    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c2617cf7-6369-4fc4-9fb5-6e6cfc4e9add-host-cni-netd\") pod \"ovnkube-node-f9qxl\" (UID: \"c2617cf7-6369-4fc4-9fb5-6e6cfc4e9add\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9qxl"
Sep 16 17:24:50.131095 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:50.130769    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c2617cf7-6369-4fc4-9fb5-6e6cfc4e9add-run-ovn\") pod \"ovnkube-node-f9qxl\" (UID: \"c2617cf7-6369-4fc4-9fb5-6e6cfc4e9add\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9qxl"
Sep 16 17:24:50.131095 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:50.130776    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5df1fb59-c460-463c-8691-2c270b69aa34-system-cni-dir\") pod \"multus-additional-cni-plugins-6fsn7\" (UID: \"5df1fb59-c460-463c-8691-2c270b69aa34\") " pod="openshift-multus/multus-additional-cni-plugins-6fsn7"
Sep 16 17:24:50.131095 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:50.130729    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c2617cf7-6369-4fc4-9fb5-6e6cfc4e9add-host-kubelet\") pod \"ovnkube-node-f9qxl\" (UID: \"c2617cf7-6369-4fc4-9fb5-6e6cfc4e9add\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9qxl"
Sep 16 17:24:50.131095 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:50.130806    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6bfc5b33-9432-43a1-a89a-8a2f18acc45f-etc-kubernetes\") pod \"tuned-pgq7f\" (UID: \"6bfc5b33-9432-43a1-a89a-8a2f18acc45f\") " pod="openshift-cluster-node-tuning-operator/tuned-pgq7f"
Sep 16 17:24:50.131095 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:50.130818    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c2617cf7-6369-4fc4-9fb5-6e6cfc4e9add-host-cni-bin\") pod \"ovnkube-node-f9qxl\" (UID: \"c2617cf7-6369-4fc4-9fb5-6e6cfc4e9add\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9qxl"
Sep 16 17:24:50.131095 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:50.130826    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/6bfc5b33-9432-43a1-a89a-8a2f18acc45f-run\") pod \"tuned-pgq7f\" (UID: \"6bfc5b33-9432-43a1-a89a-8a2f18acc45f\") " pod="openshift-cluster-node-tuning-operator/tuned-pgq7f"
Sep 16 17:24:50.131095 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:50.130871    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/41b48733-fd0e-4eaa-a91c-5d57f642b2f8-serviceca\") pod \"node-ca-t6cx6\" (UID: \"41b48733-fd0e-4eaa-a91c-5d57f642b2f8\") " pod="openshift-image-registry/node-ca-t6cx6"
Sep 16 17:24:50.131095 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:50.130886    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/6bfc5b33-9432-43a1-a89a-8a2f18acc45f-etc-modprobe-d\") pod \"tuned-pgq7f\" (UID: \"6bfc5b33-9432-43a1-a89a-8a2f18acc45f\") " pod="openshift-cluster-node-tuning-operator/tuned-pgq7f"
Sep 16 17:24:50.131095 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:50.130897    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/6bfc5b33-9432-43a1-a89a-8a2f18acc45f-run\") pod \"tuned-pgq7f\" (UID: \"6bfc5b33-9432-43a1-a89a-8a2f18acc45f\") " pod="openshift-cluster-node-tuning-operator/tuned-pgq7f"
Sep 16 17:24:50.131883 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:50.130899    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c2617cf7-6369-4fc4-9fb5-6e6cfc4e9add-ovn-node-metrics-cert\") pod \"ovnkube-node-f9qxl\" (UID: \"c2617cf7-6369-4fc4-9fb5-6e6cfc4e9add\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9qxl"
Sep 16 17:24:50.131883 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:50.130923    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6l4d\" (UniqueName: \"kubernetes.io/projected/c2617cf7-6369-4fc4-9fb5-6e6cfc4e9add-kube-api-access-n6l4d\") pod \"ovnkube-node-f9qxl\" (UID: \"c2617cf7-6369-4fc4-9fb5-6e6cfc4e9add\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9qxl"
Sep 16 17:24:50.131883 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:50.130934    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5df1fb59-c460-463c-8691-2c270b69aa34-system-cni-dir\") pod \"multus-additional-cni-plugins-6fsn7\" (UID: \"5df1fb59-c460-463c-8691-2c270b69aa34\") " pod="openshift-multus/multus-additional-cni-plugins-6fsn7"
Sep 16 17:24:50.131883 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:50.130944    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5df1fb59-c460-463c-8691-2c270b69aa34-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6fsn7\" (UID: \"5df1fb59-c460-463c-8691-2c270b69aa34\") " pod="openshift-multus/multus-additional-cni-plugins-6fsn7"
Sep 16 17:24:50.131883 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:50.130900    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/6bfc5b33-9432-43a1-a89a-8a2f18acc45f-etc-sysctl-d\") pod \"tuned-pgq7f\" (UID: \"6bfc5b33-9432-43a1-a89a-8a2f18acc45f\") " pod="openshift-cluster-node-tuning-operator/tuned-pgq7f"
Sep 16 17:24:50.131883 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:50.130970    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/6bfc5b33-9432-43a1-a89a-8a2f18acc45f-etc-sysctl-conf\") pod \"tuned-pgq7f\" (UID: \"6bfc5b33-9432-43a1-a89a-8a2f18acc45f\") " pod="openshift-cluster-node-tuning-operator/tuned-pgq7f"
Sep 16 17:24:50.131883 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:50.130949    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6bfc5b33-9432-43a1-a89a-8a2f18acc45f-etc-kubernetes\") pod \"tuned-pgq7f\" (UID: \"6bfc5b33-9432-43a1-a89a-8a2f18acc45f\") " pod="openshift-cluster-node-tuning-operator/tuned-pgq7f"
Sep 16 17:24:50.131883 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:50.130989    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6bfc5b33-9432-43a1-a89a-8a2f18acc45f-sys\") pod \"tuned-pgq7f\" (UID: \"6bfc5b33-9432-43a1-a89a-8a2f18acc45f\") " pod="openshift-cluster-node-tuning-operator/tuned-pgq7f"
Sep 16 17:24:50.131883 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:50.131010    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21a2db47-94e4-4d2b-9cb2-0b2bca64ef5b-tmp-dir\") pod \"node-resolver-qtdvz\" (UID: \"21a2db47-94e4-4d2b-9cb2-0b2bca64ef5b\") " pod="openshift-dns/node-resolver-qtdvz"
Sep 16 17:24:50.131883 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:50.131038    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e9240885-14d4-4b43-bf98-442e30d61c36-original-pull-secret\") pod \"global-pull-secret-syncer-s7zg6\" (UID: \"e9240885-14d4-4b43-bf98-442e30d61c36\") " pod="kube-system/global-pull-secret-syncer-s7zg6"
Sep 16 17:24:50.131883 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:50.131061    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c2617cf7-6369-4fc4-9fb5-6e6cfc4e9add-host-run-netns\") pod \"ovnkube-node-f9qxl\" (UID: \"c2617cf7-6369-4fc4-9fb5-6e6cfc4e9add\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9qxl"
Sep 16 17:24:50.131883 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:50.131098    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c2617cf7-6369-4fc4-9fb5-6e6cfc4e9add-log-socket\") pod \"ovnkube-node-f9qxl\" (UID: \"c2617cf7-6369-4fc4-9fb5-6e6cfc4e9add\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9qxl"
Sep 16 17:24:50.131883 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:50.131120    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hskq\" (UniqueName: \"kubernetes.io/projected/6bfc5b33-9432-43a1-a89a-8a2f18acc45f-kube-api-access-8hskq\") pod \"tuned-pgq7f\" (UID: \"6bfc5b33-9432-43a1-a89a-8a2f18acc45f\") " pod="openshift-cluster-node-tuning-operator/tuned-pgq7f"
Sep 16 17:24:50.131883 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:50.131140    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzg2d\" (UniqueName: \"kubernetes.io/projected/41b48733-fd0e-4eaa-a91c-5d57f642b2f8-kube-api-access-wzg2d\") pod \"node-ca-t6cx6\" (UID: \"41b48733-fd0e-4eaa-a91c-5d57f642b2f8\") " pod="openshift-image-registry/node-ca-t6cx6"
Sep 16 17:24:50.131883 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:50.131168    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c2617cf7-6369-4fc4-9fb5-6e6cfc4e9add-host-run-ovn-kubernetes\") pod \"ovnkube-node-f9qxl\" (UID: \"c2617cf7-6369-4fc4-9fb5-6e6cfc4e9add\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9qxl"
Sep 16 17:24:50.131883 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:50.131190    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c2617cf7-6369-4fc4-9fb5-6e6cfc4e9add-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-f9qxl\" (UID: \"c2617cf7-6369-4fc4-9fb5-6e6cfc4e9add\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9qxl"
Sep 16 17:24:50.131883 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:50.131214    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxqrk\" (UniqueName: \"kubernetes.io/projected/21a2db47-94e4-4d2b-9cb2-0b2bca64ef5b-kube-api-access-mxqrk\") pod \"node-resolver-qtdvz\" (UID: \"21a2db47-94e4-4d2b-9cb2-0b2bca64ef5b\") " pod="openshift-dns/node-resolver-qtdvz"
Sep 16 17:24:50.132676 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:50.131236    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bc719829-66b1-4a84-b994-e71cf111dafa-metrics-certs\") pod \"network-metrics-daemon-xfgjq\" (UID: \"bc719829-66b1-4a84-b994-e71cf111dafa\") " pod="openshift-multus/network-metrics-daemon-xfgjq"
Sep 16 17:24:50.132676 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:50.131248    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c2617cf7-6369-4fc4-9fb5-6e6cfc4e9add-ovnkube-config\") pod \"ovnkube-node-f9qxl\" (UID: \"c2617cf7-6369-4fc4-9fb5-6e6cfc4e9add\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9qxl"
Sep 16 17:24:50.132676 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:50.131259    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5df1fb59-c460-463c-8691-2c270b69aa34-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6fsn7\" (UID: \"5df1fb59-c460-463c-8691-2c270b69aa34\") " pod="openshift-multus/multus-additional-cni-plugins-6fsn7"
Sep 16 17:24:50.132676 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:50.131290    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6bfc5b33-9432-43a1-a89a-8a2f18acc45f-lib-modules\") pod \"tuned-pgq7f\" (UID: \"6bfc5b33-9432-43a1-a89a-8a2f18acc45f\") " pod="openshift-cluster-node-tuning-operator/tuned-pgq7f"
Sep 16 17:24:50.132676 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:50.131310    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/41b48733-fd0e-4eaa-a91c-5d57f642b2f8-serviceca\") pod \"node-ca-t6cx6\" (UID: \"41b48733-fd0e-4eaa-a91c-5d57f642b2f8\") " pod="openshift-image-registry/node-ca-t6cx6"
Sep 16 17:24:50.132676 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:50.131316    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c2617cf7-6369-4fc4-9fb5-6e6cfc4e9add-var-lib-openvswitch\") pod \"ovnkube-node-f9qxl\" (UID: \"c2617cf7-6369-4fc4-9fb5-6e6cfc4e9add\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9qxl"
Sep 16 17:24:50.132676 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:50.131338    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c2617cf7-6369-4fc4-9fb5-6e6cfc4e9add-etc-openvswitch\") pod \"ovnkube-node-f9qxl\" (UID: \"c2617cf7-6369-4fc4-9fb5-6e6cfc4e9add\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9qxl"
Sep 16 17:24:50.132676 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:50.131359    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e9240885-14d4-4b43-bf98-442e30d61c36-dbus\") pod \"global-pull-secret-syncer-s7zg6\" (UID: \"e9240885-14d4-4b43-bf98-442e30d61c36\") " pod="kube-system/global-pull-secret-syncer-s7zg6"
Sep 16 17:24:50.132676 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:50.131370    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6bfc5b33-9432-43a1-a89a-8a2f18acc45f-sys\") pod \"tuned-pgq7f\" (UID: \"6bfc5b33-9432-43a1-a89a-8a2f18acc45f\") " pod="openshift-cluster-node-tuning-operator/tuned-pgq7f"
Sep 16 17:24:50.132676 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:50.131375    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/6bfc5b33-9432-43a1-a89a-8a2f18acc45f-etc-tuned\") pod \"tuned-pgq7f\" (UID: \"6bfc5b33-9432-43a1-a89a-8a2f18acc45f\") " pod="openshift-cluster-node-tuning-operator/tuned-pgq7f"
Sep 16 17:24:50.132676 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:50.131435    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/6bfc5b33-9432-43a1-a89a-8a2f18acc45f-etc-sysctl-conf\") pod \"tuned-pgq7f\" (UID: \"6bfc5b33-9432-43a1-a89a-8a2f18acc45f\") " pod="openshift-cluster-node-tuning-operator/tuned-pgq7f"
Sep 16 17:24:50.132676 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:50.131491    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5df1fb59-c460-463c-8691-2c270b69aa34-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6fsn7\" (UID: \"5df1fb59-c460-463c-8691-2c270b69aa34\") " pod="openshift-multus/multus-additional-cni-plugins-6fsn7"
Sep 16 17:24:50.132676 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:50.131548    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c2617cf7-6369-4fc4-9fb5-6e6cfc4e9add-etc-openvswitch\") pod \"ovnkube-node-f9qxl\" (UID: \"c2617cf7-6369-4fc4-9fb5-6e6cfc4e9add\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9qxl"
Sep 16 17:24:50.132676 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:50.131579    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5df1fb59-c460-463c-8691-2c270b69aa34-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6fsn7\" (UID: \"5df1fb59-c460-463c-8691-2c270b69aa34\") " pod="openshift-multus/multus-additional-cni-plugins-6fsn7"
Sep 16 17:24:50.132676 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:50.131589    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e9240885-14d4-4b43-bf98-442e30d61c36-dbus\") pod \"global-pull-secret-syncer-s7zg6\" (UID: \"e9240885-14d4-4b43-bf98-442e30d61c36\") " pod="kube-system/global-pull-secret-syncer-s7zg6"
Sep 16 17:24:50.132676 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:50.131624    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6bfc5b33-9432-43a1-a89a-8a2f18acc45f-lib-modules\") pod \"tuned-pgq7f\" (UID: \"6bfc5b33-9432-43a1-a89a-8a2f18acc45f\") " pod="openshift-cluster-node-tuning-operator/tuned-pgq7f"
Sep 16 17:24:50.132676 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:50.131639    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21a2db47-94e4-4d2b-9cb2-0b2bca64ef5b-tmp-dir\") pod \"node-resolver-qtdvz\" (UID: \"21a2db47-94e4-4d2b-9cb2-0b2bca64ef5b\") " pod="openshift-dns/node-resolver-qtdvz"
Sep 16 17:24:50.132676 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:50.131649    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c2617cf7-6369-4fc4-9fb5-6e6cfc4e9add-host-run-netns\") pod \"ovnkube-node-f9qxl\" (UID: \"c2617cf7-6369-4fc4-9fb5-6e6cfc4e9add\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9qxl"
Sep 16 17:24:50.133426 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:50.131638    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c2617cf7-6369-4fc4-9fb5-6e6cfc4e9add-var-lib-openvswitch\") pod \"ovnkube-node-f9qxl\" (UID: \"c2617cf7-6369-4fc4-9fb5-6e6cfc4e9add\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9qxl"
Sep 16 17:24:50.133426 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:50.131693    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c2617cf7-6369-4fc4-9fb5-6e6cfc4e9add-log-socket\") pod \"ovnkube-node-f9qxl\" (UID: \"c2617cf7-6369-4fc4-9fb5-6e6cfc4e9add\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9qxl"
Sep 16 17:24:50.133426 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:50.131726    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c2617cf7-6369-4fc4-9fb5-6e6cfc4e9add-host-run-ovn-kubernetes\") pod \"ovnkube-node-f9qxl\" (UID: \"c2617cf7-6369-4fc4-9fb5-6e6cfc4e9add\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9qxl"
Sep 16 17:24:50.133426 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:50.131728    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c2617cf7-6369-4fc4-9fb5-6e6cfc4e9add-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-f9qxl\" (UID: \"c2617cf7-6369-4fc4-9fb5-6e6cfc4e9add\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9qxl"
Sep 16 17:24:50.133426 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:24:50.131747    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 17:24:50.133426 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:24:50.131802    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 17:24:50.133426 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:24:50.131823    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e9240885-14d4-4b43-bf98-442e30d61c36-original-pull-secret podName:e9240885-14d4-4b43-bf98-442e30d61c36 nodeName:}" failed. No retries permitted until 2026-09-16 17:24:50.631804599 +0000 UTC m=+3.101802732 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e9240885-14d4-4b43-bf98-442e30d61c36-original-pull-secret") pod "global-pull-secret-syncer-s7zg6" (UID: "e9240885-14d4-4b43-bf98-442e30d61c36") : object "kube-system"/"original-pull-secret" not registered
Sep 16 17:24:50.133426 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:24:50.131862    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc719829-66b1-4a84-b994-e71cf111dafa-metrics-certs podName:bc719829-66b1-4a84-b994-e71cf111dafa nodeName:}" failed. No retries permitted until 2026-09-16 17:24:50.631832596 +0000 UTC m=+3.101830715 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bc719829-66b1-4a84-b994-e71cf111dafa-metrics-certs") pod "network-metrics-daemon-xfgjq" (UID: "bc719829-66b1-4a84-b994-e71cf111dafa") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 17:24:50.133426 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:50.131879    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c2617cf7-6369-4fc4-9fb5-6e6cfc4e9add-run-systemd\") pod \"ovnkube-node-f9qxl\" (UID: \"c2617cf7-6369-4fc4-9fb5-6e6cfc4e9add\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9qxl"
Sep 16 17:24:50.133426 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:50.131904    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e9240885-14d4-4b43-bf98-442e30d61c36-kubelet-config\") pod \"global-pull-secret-syncer-s7zg6\" (UID: \"e9240885-14d4-4b43-bf98-442e30d61c36\") " pod="kube-system/global-pull-secret-syncer-s7zg6"
Sep 16 17:24:50.133426 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:50.131925    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5df1fb59-c460-463c-8691-2c270b69aa34-cni-binary-copy\") pod \"multus-additional-cni-plugins-6fsn7\" (UID: \"5df1fb59-c460-463c-8691-2c270b69aa34\") " pod="openshift-multus/multus-additional-cni-plugins-6fsn7"
Sep 16 17:24:50.133426 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:50.131946    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c2617cf7-6369-4fc4-9fb5-6e6cfc4e9add-host-slash\") pod \"ovnkube-node-f9qxl\" (UID: \"c2617cf7-6369-4fc4-9fb5-6e6cfc4e9add\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9qxl"
Sep 16 17:24:50.133426 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:50.131956    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c2617cf7-6369-4fc4-9fb5-6e6cfc4e9add-run-systemd\") pod \"ovnkube-node-f9qxl\" (UID: \"c2617cf7-6369-4fc4-9fb5-6e6cfc4e9add\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9qxl"
Sep 16 17:24:50.133426 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:50.131967    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e9240885-14d4-4b43-bf98-442e30d61c36-kubelet-config\") pod \"global-pull-secret-syncer-s7zg6\" (UID: \"e9240885-14d4-4b43-bf98-442e30d61c36\") " pod="kube-system/global-pull-secret-syncer-s7zg6"
Sep 16 17:24:50.133426 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:50.131961    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/6bfc5b33-9432-43a1-a89a-8a2f18acc45f-etc-systemd\") pod \"tuned-pgq7f\" (UID: \"6bfc5b33-9432-43a1-a89a-8a2f18acc45f\") " pod="openshift-cluster-node-tuning-operator/tuned-pgq7f"
Sep 16 17:24:50.133426 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:50.131987    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/6bfc5b33-9432-43a1-a89a-8a2f18acc45f-etc-systemd\") pod \"tuned-pgq7f\" (UID: \"6bfc5b33-9432-43a1-a89a-8a2f18acc45f\") " pod="openshift-cluster-node-tuning-operator/tuned-pgq7f"
Sep 16 17:24:50.134022 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:50.132012    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chp67\" (UniqueName: \"kubernetes.io/projected/5df1fb59-c460-463c-8691-2c270b69aa34-kube-api-access-chp67\") pod \"multus-additional-cni-plugins-6fsn7\" (UID: \"5df1fb59-c460-463c-8691-2c270b69aa34\") " pod="openshift-multus/multus-additional-cni-plugins-6fsn7"
Sep 16 17:24:50.134022 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:50.132042    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c2617cf7-6369-4fc4-9fb5-6e6cfc4e9add-systemd-units\") pod \"ovnkube-node-f9qxl\" (UID: \"c2617cf7-6369-4fc4-9fb5-6e6cfc4e9add\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9qxl"
Sep 16 17:24:50.134022 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:50.132064    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c2617cf7-6369-4fc4-9fb5-6e6cfc4e9add-node-log\") pod \"ovnkube-node-f9qxl\" (UID: \"c2617cf7-6369-4fc4-9fb5-6e6cfc4e9add\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9qxl"
Sep 16 17:24:50.134022 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:50.132074    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c2617cf7-6369-4fc4-9fb5-6e6cfc4e9add-host-slash\") pod \"ovnkube-node-f9qxl\" (UID: \"c2617cf7-6369-4fc4-9fb5-6e6cfc4e9add\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9qxl"
Sep 16 17:24:50.134022 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:50.132086    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c2617cf7-6369-4fc4-9fb5-6e6cfc4e9add-env-overrides\") pod \"ovnkube-node-f9qxl\" (UID: \"c2617cf7-6369-4fc4-9fb5-6e6cfc4e9add\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9qxl"
Sep 16 17:24:50.134022 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:50.132110    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmzzm\" (UniqueName: \"kubernetes.io/projected/bc719829-66b1-4a84-b994-e71cf111dafa-kube-api-access-mmzzm\") pod \"network-metrics-daemon-xfgjq\" (UID: \"bc719829-66b1-4a84-b994-e71cf111dafa\") " pod="openshift-multus/network-metrics-daemon-xfgjq"
Sep 16 17:24:50.134022 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:50.132118    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c2617cf7-6369-4fc4-9fb5-6e6cfc4e9add-systemd-units\") pod \"ovnkube-node-f9qxl\" (UID: \"c2617cf7-6369-4fc4-9fb5-6e6cfc4e9add\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9qxl"
Sep 16 17:24:50.134022 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:50.132132    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x46mt\" (UniqueName: \"kubernetes.io/projected/1d620b9d-0055-4ed6-aced-2f4cfc54cbd8-kube-api-access-x46mt\") pod \"network-check-target-4c79x\" (UID: \"1d620b9d-0055-4ed6-aced-2f4cfc54cbd8\") " pod="openshift-network-diagnostics/network-check-target-4c79x"
Sep 16 17:24:50.134022 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:50.132158    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c2617cf7-6369-4fc4-9fb5-6e6cfc4e9add-run-openvswitch\") pod \"ovnkube-node-f9qxl\" (UID: \"c2617cf7-6369-4fc4-9fb5-6e6cfc4e9add\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9qxl"
Sep 16 17:24:50.134022 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:50.132186    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c2617cf7-6369-4fc4-9fb5-6e6cfc4e9add-ovnkube-script-lib\") pod \"ovnkube-node-f9qxl\" (UID: \"c2617cf7-6369-4fc4-9fb5-6e6cfc4e9add\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9qxl"
Sep 16 17:24:50.134022 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:50.132213    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5df1fb59-c460-463c-8691-2c270b69aa34-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6fsn7\" (UID: \"5df1fb59-c460-463c-8691-2c270b69aa34\") " pod="openshift-multus/multus-additional-cni-plugins-6fsn7"
Sep 16 17:24:50.134022 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:50.132236    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6bfc5b33-9432-43a1-a89a-8a2f18acc45f-tmp\") pod \"tuned-pgq7f\" (UID: \"6bfc5b33-9432-43a1-a89a-8a2f18acc45f\") " pod="openshift-cluster-node-tuning-operator/tuned-pgq7f"
Sep 16 17:24:50.134022 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:50.132249    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c2617cf7-6369-4fc4-9fb5-6e6cfc4e9add-node-log\") pod \"ovnkube-node-f9qxl\" (UID: \"c2617cf7-6369-4fc4-9fb5-6e6cfc4e9add\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9qxl"
Sep 16 17:24:50.134022 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:50.132263    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6bfc5b33-9432-43a1-a89a-8a2f18acc45f-var-lib-kubelet\") pod \"tuned-pgq7f\" (UID: \"6bfc5b33-9432-43a1-a89a-8a2f18acc45f\") " pod="openshift-cluster-node-tuning-operator/tuned-pgq7f"
Sep 16 17:24:50.134022 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:50.132288    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c2617cf7-6369-4fc4-9fb5-6e6cfc4e9add-run-openvswitch\") pod \"ovnkube-node-f9qxl\" (UID: \"c2617cf7-6369-4fc4-9fb5-6e6cfc4e9add\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9qxl"
Sep 16 17:24:50.134022 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:50.132342    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6bfc5b33-9432-43a1-a89a-8a2f18acc45f-var-lib-kubelet\") pod \"tuned-pgq7f\" (UID: \"6bfc5b33-9432-43a1-a89a-8a2f18acc45f\") " pod="openshift-cluster-node-tuning-operator/tuned-pgq7f"
Sep 16 17:24:50.134022 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:50.132392    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5df1fb59-c460-463c-8691-2c270b69aa34-cni-binary-copy\") pod \"multus-additional-cni-plugins-6fsn7\" (UID: \"5df1fb59-c460-463c-8691-2c270b69aa34\") " pod="openshift-multus/multus-additional-cni-plugins-6fsn7"
Sep 16 17:24:50.134714 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:50.132462    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c2617cf7-6369-4fc4-9fb5-6e6cfc4e9add-env-overrides\") pod \"ovnkube-node-f9qxl\" (UID: \"c2617cf7-6369-4fc4-9fb5-6e6cfc4e9add\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9qxl"
Sep 16 17:24:50.134714 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:50.132818    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5df1fb59-c460-463c-8691-2c270b69aa34-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6fsn7\" (UID: \"5df1fb59-c460-463c-8691-2c270b69aa34\") " pod="openshift-multus/multus-additional-cni-plugins-6fsn7"
Sep 16 17:24:50.134714 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:50.132931    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c2617cf7-6369-4fc4-9fb5-6e6cfc4e9add-ovnkube-script-lib\") pod \"ovnkube-node-f9qxl\" (UID: \"c2617cf7-6369-4fc4-9fb5-6e6cfc4e9add\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9qxl"
Sep 16 17:24:50.134714 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:50.133502    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c2617cf7-6369-4fc4-9fb5-6e6cfc4e9add-ovn-node-metrics-cert\") pod \"ovnkube-node-f9qxl\" (UID: \"c2617cf7-6369-4fc4-9fb5-6e6cfc4e9add\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9qxl"
Sep 16 17:24:50.134714 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:50.134247    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/6bfc5b33-9432-43a1-a89a-8a2f18acc45f-etc-tuned\") pod \"tuned-pgq7f\" (UID: \"6bfc5b33-9432-43a1-a89a-8a2f18acc45f\") " pod="openshift-cluster-node-tuning-operator/tuned-pgq7f"
Sep 16 17:24:50.134922 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:50.134786    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6bfc5b33-9432-43a1-a89a-8a2f18acc45f-tmp\") pod \"tuned-pgq7f\" (UID: \"6bfc5b33-9432-43a1-a89a-8a2f18acc45f\") " pod="openshift-cluster-node-tuning-operator/tuned-pgq7f"
Sep 16 17:24:50.151421 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:50.151389    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6l4d\" (UniqueName: \"kubernetes.io/projected/c2617cf7-6369-4fc4-9fb5-6e6cfc4e9add-kube-api-access-n6l4d\") pod \"ovnkube-node-f9qxl\" (UID: \"c2617cf7-6369-4fc4-9fb5-6e6cfc4e9add\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9qxl"
Sep 16 17:24:50.160517 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:50.160484    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzg2d\" (UniqueName: \"kubernetes.io/projected/41b48733-fd0e-4eaa-a91c-5d57f642b2f8-kube-api-access-wzg2d\") pod \"node-ca-t6cx6\" (UID: \"41b48733-fd0e-4eaa-a91c-5d57f642b2f8\") " pod="openshift-image-registry/node-ca-t6cx6"
Sep 16 17:24:50.165875 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:50.165834    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-chp67\" (UniqueName: \"kubernetes.io/projected/5df1fb59-c460-463c-8691-2c270b69aa34-kube-api-access-chp67\") pod \"multus-additional-cni-plugins-6fsn7\" (UID: \"5df1fb59-c460-463c-8691-2c270b69aa34\") " pod="openshift-multus/multus-additional-cni-plugins-6fsn7"
Sep 16 17:24:50.173776 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:24:50.173720    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 17:24:50.173776 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:24:50.173749    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 17:24:50.173776 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:24:50.173763    2579 projected.go:196] Error preparing data for projected volume kube-api-access-x46mt for pod openshift-network-diagnostics/network-check-target-4c79x: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 17:24:50.174053 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:24:50.173836    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1d620b9d-0055-4ed6-aced-2f4cfc54cbd8-kube-api-access-x46mt podName:1d620b9d-0055-4ed6-aced-2f4cfc54cbd8 nodeName:}" failed. No retries permitted until 2026-09-16 17:24:50.673809978 +0000 UTC m=+3.143808098 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-x46mt" (UniqueName: "kubernetes.io/projected/1d620b9d-0055-4ed6-aced-2f4cfc54cbd8-kube-api-access-x46mt") pod "network-check-target-4c79x" (UID: "1d620b9d-0055-4ed6-aced-2f4cfc54cbd8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 17:24:50.175201 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:50.175168    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxqrk\" (UniqueName: \"kubernetes.io/projected/21a2db47-94e4-4d2b-9cb2-0b2bca64ef5b-kube-api-access-mxqrk\") pod \"node-resolver-qtdvz\" (UID: \"21a2db47-94e4-4d2b-9cb2-0b2bca64ef5b\") " pod="openshift-dns/node-resolver-qtdvz"
Sep 16 17:24:50.176544 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:50.176526    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmzzm\" (UniqueName: \"kubernetes.io/projected/bc719829-66b1-4a84-b994-e71cf111dafa-kube-api-access-mmzzm\") pod \"network-metrics-daemon-xfgjq\" (UID: \"bc719829-66b1-4a84-b994-e71cf111dafa\") " pod="openshift-multus/network-metrics-daemon-xfgjq"
Sep 16 17:24:50.176712 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:50.176691    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hskq\" (UniqueName: \"kubernetes.io/projected/6bfc5b33-9432-43a1-a89a-8a2f18acc45f-kube-api-access-8hskq\") pod \"tuned-pgq7f\" (UID: \"6bfc5b33-9432-43a1-a89a-8a2f18acc45f\") " pod="openshift-cluster-node-tuning-operator/tuned-pgq7f"
Sep 16 17:24:50.224790 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:50.224757    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-gmmwp"
Sep 16 17:24:50.232896 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:50.232868    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w58mp"
Sep 16 17:24:50.244299 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:50.244268    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-gntnz"
Sep 16 17:24:50.249324 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:50.249305    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-59rw5"
Sep 16 17:24:50.257983 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:50.257961    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-pgq7f"
Sep 16 17:24:50.264378 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:50.264357    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-t6cx6"
Sep 16 17:24:50.272937 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:50.272918    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-6fsn7"
Sep 16 17:24:50.278555 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:50.278532    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-f9qxl"
Sep 16 17:24:50.284669 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:50.284649    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-qtdvz"
Sep 16 17:24:50.495454 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:50.495376    2579 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 17:24:50.636008 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:50.635972    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e9240885-14d4-4b43-bf98-442e30d61c36-original-pull-secret\") pod \"global-pull-secret-syncer-s7zg6\" (UID: \"e9240885-14d4-4b43-bf98-442e30d61c36\") " pod="kube-system/global-pull-secret-syncer-s7zg6"
Sep 16 17:24:50.636008 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:50.636024    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bc719829-66b1-4a84-b994-e71cf111dafa-metrics-certs\") pod \"network-metrics-daemon-xfgjq\" (UID: \"bc719829-66b1-4a84-b994-e71cf111dafa\") " pod="openshift-multus/network-metrics-daemon-xfgjq"
Sep 16 17:24:50.636244 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:24:50.636144    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 17:24:50.636244 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:24:50.636157    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 17:24:50.636244 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:24:50.636209    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc719829-66b1-4a84-b994-e71cf111dafa-metrics-certs podName:bc719829-66b1-4a84-b994-e71cf111dafa nodeName:}" failed. No retries permitted until 2026-09-16 17:24:51.636196188 +0000 UTC m=+4.106194316 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bc719829-66b1-4a84-b994-e71cf111dafa-metrics-certs") pod "network-metrics-daemon-xfgjq" (UID: "bc719829-66b1-4a84-b994-e71cf111dafa") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 17:24:50.636244 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:24:50.636226    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e9240885-14d4-4b43-bf98-442e30d61c36-original-pull-secret podName:e9240885-14d4-4b43-bf98-442e30d61c36 nodeName:}" failed. No retries permitted until 2026-09-16 17:24:51.636219514 +0000 UTC m=+4.106217643 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e9240885-14d4-4b43-bf98-442e30d61c36-original-pull-secret") pod "global-pull-secret-syncer-s7zg6" (UID: "e9240885-14d4-4b43-bf98-442e30d61c36") : object "kube-system"/"original-pull-secret" not registered
Sep 16 17:24:50.737063 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:50.737028    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x46mt\" (UniqueName: \"kubernetes.io/projected/1d620b9d-0055-4ed6-aced-2f4cfc54cbd8-kube-api-access-x46mt\") pod \"network-check-target-4c79x\" (UID: \"1d620b9d-0055-4ed6-aced-2f4cfc54cbd8\") " pod="openshift-network-diagnostics/network-check-target-4c79x"
Sep 16 17:24:50.737251 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:24:50.737223    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 17:24:50.737317 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:24:50.737253    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 17:24:50.737317 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:24:50.737266    2579 projected.go:196] Error preparing data for projected volume kube-api-access-x46mt for pod openshift-network-diagnostics/network-check-target-4c79x: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 17:24:50.737412 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:24:50.737318    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1d620b9d-0055-4ed6-aced-2f4cfc54cbd8-kube-api-access-x46mt podName:1d620b9d-0055-4ed6-aced-2f4cfc54cbd8 nodeName:}" failed. No retries permitted until 2026-09-16 17:24:51.737304051 +0000 UTC m=+4.207302177 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-x46mt" (UniqueName: "kubernetes.io/projected/1d620b9d-0055-4ed6-aced-2f4cfc54cbd8-kube-api-access-x46mt") pod "network-check-target-4c79x" (UID: "1d620b9d-0055-4ed6-aced-2f4cfc54cbd8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 17:24:50.851066 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:50.851011    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5df1fb59_c460_463c_8691_2c270b69aa34.slice/crio-10e54208d5ff7f22ce929a63b11050411494f548a48ec6f6b44e49d46c3a915d WatchSource:0}: Error finding container 10e54208d5ff7f22ce929a63b11050411494f548a48ec6f6b44e49d46c3a915d: Status 404 returned error can't find the container with id 10e54208d5ff7f22ce929a63b11050411494f548a48ec6f6b44e49d46c3a915d
Sep 16 17:24:50.854171 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:50.854148    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bfc5b33_9432_43a1_a89a_8a2f18acc45f.slice/crio-566cb5b1ce129c6e397d4fb3566426d06d826ebdec710f51ad49e5041f52829a WatchSource:0}: Error finding container 566cb5b1ce129c6e397d4fb3566426d06d826ebdec710f51ad49e5041f52829a: Status 404 returned error can't find the container with id 566cb5b1ce129c6e397d4fb3566426d06d826ebdec710f51ad49e5041f52829a
Sep 16 17:24:50.855890 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:50.855860    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef27aa30_cb45_45bc_8d0e_50107fc6459d.slice/crio-607c6eb065b8911047dcacce986f5b04ec2e6da874e1c07110512cdfdce506d4 WatchSource:0}: Error finding container 607c6eb065b8911047dcacce986f5b04ec2e6da874e1c07110512cdfdce506d4: Status 404 returned error can't find the container with id 607c6eb065b8911047dcacce986f5b04ec2e6da874e1c07110512cdfdce506d4
Sep 16 17:24:50.856504 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:50.856326    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ea5d21d_1046_4edb_b1d9_9ca43ebc564d.slice/crio-f5f6220118437284057fe49b09cdf42bc620145380c3bd4ad30162d116a9b26a WatchSource:0}: Error finding container f5f6220118437284057fe49b09cdf42bc620145380c3bd4ad30162d116a9b26a: Status 404 returned error can't find the container with id f5f6220118437284057fe49b09cdf42bc620145380c3bd4ad30162d116a9b26a
Sep 16 17:24:50.857078 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:50.857017    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2617cf7_6369_4fc4_9fb5_6e6cfc4e9add.slice/crio-3cf96de50b2929fff116fa67d2e748bd0390f5539ee47d156fdf9d6452843a9f WatchSource:0}: Error finding container 3cf96de50b2929fff116fa67d2e748bd0390f5539ee47d156fdf9d6452843a9f: Status 404 returned error can't find the container with id 3cf96de50b2929fff116fa67d2e748bd0390f5539ee47d156fdf9d6452843a9f
Sep 16 17:24:50.858705 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:50.858603    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1223407_0c9e_4ed6_bd17_cf0f3962e27e.slice/crio-6b6b18b8d8c93c691cc966d8dafc69beeddb5350867fec44253d6c2c11a30214 WatchSource:0}: Error finding container 6b6b18b8d8c93c691cc966d8dafc69beeddb5350867fec44253d6c2c11a30214: Status 404 returned error can't find the container with id 6b6b18b8d8c93c691cc966d8dafc69beeddb5350867fec44253d6c2c11a30214
Sep 16 17:24:50.859955 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:50.859921    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41b48733_fd0e_4eaa_a91c_5d57f642b2f8.slice/crio-1af6296f208bc0c4b95ff4cbdce7d91a35fdac4661a0453714ed6e75d6f6fffd WatchSource:0}: Error finding container 1af6296f208bc0c4b95ff4cbdce7d91a35fdac4661a0453714ed6e75d6f6fffd: Status 404 returned error can't find the container with id 1af6296f208bc0c4b95ff4cbdce7d91a35fdac4661a0453714ed6e75d6f6fffd
Sep 16 17:24:50.863658 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:50.863240    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc73a8482_2abb_407e_972d_05a54828be7a.slice/crio-d22aad649a6bf6afb7fb087aae0288c3a65cc120ac997afdadcfa30aaed4a543 WatchSource:0}: Error finding container d22aad649a6bf6afb7fb087aae0288c3a65cc120ac997afdadcfa30aaed4a543: Status 404 returned error can't find the container with id d22aad649a6bf6afb7fb087aae0288c3a65cc120ac997afdadcfa30aaed4a543
Sep 16 17:24:50.864215 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:24:50.864078    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21a2db47_94e4_4d2b_9cb2_0b2bca64ef5b.slice/crio-4f7c7518c8a075ca15979c778ecfd6afe9b83fd5fa8a085a8d58c6ebe656afb9 WatchSource:0}: Error finding container 4f7c7518c8a075ca15979c778ecfd6afe9b83fd5fa8a085a8d58c6ebe656afb9: Status 404 returned error can't find the container with id 4f7c7518c8a075ca15979c778ecfd6afe9b83fd5fa8a085a8d58c6ebe656afb9
Sep 16 17:24:51.044683 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:51.044644    2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-15 17:19:49 +0000 UTC" deadline="2028-05-01 11:46:46.593964635 +0000 UTC"
Sep 16 17:24:51.044683 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:51.044679    2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14226h21m55.549289507s"
Sep 16 17:24:51.093717 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:51.093688    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xfgjq"
Sep 16 17:24:51.093833 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:51.093721    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s7zg6"
Sep 16 17:24:51.093902 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:24:51.093875    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xfgjq" podUID="bc719829-66b1-4a84-b994-e71cf111dafa"
Sep 16 17:24:51.093956 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:24:51.093920    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s7zg6" podUID="e9240885-14d4-4b43-bf98-442e30d61c36"
Sep 16 17:24:51.101271 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:51.101244    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w58mp" event={"ID":"c73a8482-2abb-407e-972d-05a54828be7a","Type":"ContainerStarted","Data":"d22aad649a6bf6afb7fb087aae0288c3a65cc120ac997afdadcfa30aaed4a543"}
Sep 16 17:24:51.102309 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:51.102283    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-t6cx6" event={"ID":"41b48733-fd0e-4eaa-a91c-5d57f642b2f8","Type":"ContainerStarted","Data":"1af6296f208bc0c4b95ff4cbdce7d91a35fdac4661a0453714ed6e75d6f6fffd"}
Sep 16 17:24:51.103414 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:51.103396    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-gmmwp" event={"ID":"1ea5d21d-1046-4edb-b1d9-9ca43ebc564d","Type":"ContainerStarted","Data":"f5f6220118437284057fe49b09cdf42bc620145380c3bd4ad30162d116a9b26a"}
Sep 16 17:24:51.104428 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:51.104403    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-qtdvz" event={"ID":"21a2db47-94e4-4d2b-9cb2-0b2bca64ef5b","Type":"ContainerStarted","Data":"4f7c7518c8a075ca15979c778ecfd6afe9b83fd5fa8a085a8d58c6ebe656afb9"}
Sep 16 17:24:51.105486 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:51.105465    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-59rw5" event={"ID":"a1223407-0c9e-4ed6-bd17-cf0f3962e27e","Type":"ContainerStarted","Data":"6b6b18b8d8c93c691cc966d8dafc69beeddb5350867fec44253d6c2c11a30214"}
Sep 16 17:24:51.106601 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:51.106577    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-f9qxl" event={"ID":"c2617cf7-6369-4fc4-9fb5-6e6cfc4e9add","Type":"ContainerStarted","Data":"3cf96de50b2929fff116fa67d2e748bd0390f5539ee47d156fdf9d6452843a9f"}
Sep 16 17:24:51.107633 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:51.107613    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-gntnz" event={"ID":"ef27aa30-cb45-45bc-8d0e-50107fc6459d","Type":"ContainerStarted","Data":"607c6eb065b8911047dcacce986f5b04ec2e6da874e1c07110512cdfdce506d4"}
Sep 16 17:24:51.108886 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:51.108865    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-pgq7f" event={"ID":"6bfc5b33-9432-43a1-a89a-8a2f18acc45f","Type":"ContainerStarted","Data":"566cb5b1ce129c6e397d4fb3566426d06d826ebdec710f51ad49e5041f52829a"}
Sep 16 17:24:51.109857 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:51.109821    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6fsn7" event={"ID":"5df1fb59-c460-463c-8691-2c270b69aa34","Type":"ContainerStarted","Data":"10e54208d5ff7f22ce929a63b11050411494f548a48ec6f6b44e49d46c3a915d"}
Sep 16 17:24:51.111597 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:51.111576    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-73.ec2.internal" event={"ID":"ffbb989cf9bf70887695397b1f4835f5","Type":"ContainerStarted","Data":"1143a396fa7d5a14101a03a33f501668187b7a9c114e4cf6c27dcc30255cc5a8"}
Sep 16 17:24:51.138545 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:51.138497    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-73.ec2.internal" podStartSLOduration=3.138480545 podStartE2EDuration="3.138480545s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 17:24:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:24:51.138470004 +0000 UTC m=+3.608468165" watchObservedRunningTime="2026-09-16 17:24:51.138480545 +0000 UTC m=+3.608478709"
Sep 16 17:24:51.643023 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:51.642978    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e9240885-14d4-4b43-bf98-442e30d61c36-original-pull-secret\") pod \"global-pull-secret-syncer-s7zg6\" (UID: \"e9240885-14d4-4b43-bf98-442e30d61c36\") " pod="kube-system/global-pull-secret-syncer-s7zg6"
Sep 16 17:24:51.643201 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:51.643045    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bc719829-66b1-4a84-b994-e71cf111dafa-metrics-certs\") pod \"network-metrics-daemon-xfgjq\" (UID: \"bc719829-66b1-4a84-b994-e71cf111dafa\") " pod="openshift-multus/network-metrics-daemon-xfgjq"
Sep 16 17:24:51.643201 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:24:51.643124    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 17:24:51.643323 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:24:51.643196    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 17:24:51.643323 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:24:51.643205    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e9240885-14d4-4b43-bf98-442e30d61c36-original-pull-secret podName:e9240885-14d4-4b43-bf98-442e30d61c36 nodeName:}" failed. No retries permitted until 2026-09-16 17:24:53.643184837 +0000 UTC m=+6.113182961 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e9240885-14d4-4b43-bf98-442e30d61c36-original-pull-secret") pod "global-pull-secret-syncer-s7zg6" (UID: "e9240885-14d4-4b43-bf98-442e30d61c36") : object "kube-system"/"original-pull-secret" not registered
Sep 16 17:24:51.643323 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:24:51.643282    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc719829-66b1-4a84-b994-e71cf111dafa-metrics-certs podName:bc719829-66b1-4a84-b994-e71cf111dafa nodeName:}" failed. No retries permitted until 2026-09-16 17:24:53.643247493 +0000 UTC m=+6.113245620 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bc719829-66b1-4a84-b994-e71cf111dafa-metrics-certs") pod "network-metrics-daemon-xfgjq" (UID: "bc719829-66b1-4a84-b994-e71cf111dafa") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 17:24:51.747041 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:51.743756    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x46mt\" (UniqueName: \"kubernetes.io/projected/1d620b9d-0055-4ed6-aced-2f4cfc54cbd8-kube-api-access-x46mt\") pod \"network-check-target-4c79x\" (UID: \"1d620b9d-0055-4ed6-aced-2f4cfc54cbd8\") " pod="openshift-network-diagnostics/network-check-target-4c79x"
Sep 16 17:24:51.747041 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:24:51.743993    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 17:24:51.747041 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:24:51.744017    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 17:24:51.747041 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:24:51.744031    2579 projected.go:196] Error preparing data for projected volume kube-api-access-x46mt for pod openshift-network-diagnostics/network-check-target-4c79x: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 17:24:51.747041 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:24:51.744091    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1d620b9d-0055-4ed6-aced-2f4cfc54cbd8-kube-api-access-x46mt podName:1d620b9d-0055-4ed6-aced-2f4cfc54cbd8 nodeName:}" failed. No retries permitted until 2026-09-16 17:24:53.744072565 +0000 UTC m=+6.214070693 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-x46mt" (UniqueName: "kubernetes.io/projected/1d620b9d-0055-4ed6-aced-2f4cfc54cbd8-kube-api-access-x46mt") pod "network-check-target-4c79x" (UID: "1d620b9d-0055-4ed6-aced-2f4cfc54cbd8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 17:24:52.045047 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:52.044942    2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-15 17:19:49 +0000 UTC" deadline="2028-07-01 09:43:47.00477961 +0000 UTC"
Sep 16 17:24:52.045047 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:52.044996    2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15688h18m54.959787181s"
Sep 16 17:24:52.049109 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:52.049082    2579 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 17:24:52.096251 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:52.096218    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4c79x"
Sep 16 17:24:52.096426 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:24:52.096342    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4c79x" podUID="1d620b9d-0055-4ed6-aced-2f4cfc54cbd8"
Sep 16 17:24:52.128501 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:52.127258    2579 generic.go:356] "Generic (PLEG): container finished" podID="14dbc6cf811b53cc1d9f6ac780fa88ac" containerID="821e128332ec159a34bb1bbded26de47b3ada538c1afee42208a3ba0b9c415fa" exitCode=0
Sep 16 17:24:52.128501 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:52.128278    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-73.ec2.internal" event={"ID":"14dbc6cf811b53cc1d9f6ac780fa88ac","Type":"ContainerDied","Data":"821e128332ec159a34bb1bbded26de47b3ada538c1afee42208a3ba0b9c415fa"}
Sep 16 17:24:53.094063 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:53.093778    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xfgjq"
Sep 16 17:24:53.094525 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:24:53.094188    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xfgjq" podUID="bc719829-66b1-4a84-b994-e71cf111dafa"
Sep 16 17:24:53.094588 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:53.093954    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s7zg6"
Sep 16 17:24:53.094668 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:24:53.094645    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s7zg6" podUID="e9240885-14d4-4b43-bf98-442e30d61c36"
Sep 16 17:24:53.162386 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:53.161503    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-73.ec2.internal" event={"ID":"14dbc6cf811b53cc1d9f6ac780fa88ac","Type":"ContainerStarted","Data":"7ce066041cd24374c1b9e9134de4db5bb4cff4b107a939bf197016a80417a042"}
Sep 16 17:24:53.662229 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:53.662133    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e9240885-14d4-4b43-bf98-442e30d61c36-original-pull-secret\") pod \"global-pull-secret-syncer-s7zg6\" (UID: \"e9240885-14d4-4b43-bf98-442e30d61c36\") " pod="kube-system/global-pull-secret-syncer-s7zg6"
Sep 16 17:24:53.662229 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:53.662190    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bc719829-66b1-4a84-b994-e71cf111dafa-metrics-certs\") pod \"network-metrics-daemon-xfgjq\" (UID: \"bc719829-66b1-4a84-b994-e71cf111dafa\") " pod="openshift-multus/network-metrics-daemon-xfgjq"
Sep 16 17:24:53.662445 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:24:53.662371    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 17:24:53.662445 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:24:53.662435    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc719829-66b1-4a84-b994-e71cf111dafa-metrics-certs podName:bc719829-66b1-4a84-b994-e71cf111dafa nodeName:}" failed. No retries permitted until 2026-09-16 17:24:57.662418295 +0000 UTC m=+10.132416409 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bc719829-66b1-4a84-b994-e71cf111dafa-metrics-certs") pod "network-metrics-daemon-xfgjq" (UID: "bc719829-66b1-4a84-b994-e71cf111dafa") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 17:24:53.662893 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:24:53.662874    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 17:24:53.662991 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:24:53.662930    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e9240885-14d4-4b43-bf98-442e30d61c36-original-pull-secret podName:e9240885-14d4-4b43-bf98-442e30d61c36 nodeName:}" failed. No retries permitted until 2026-09-16 17:24:57.662914176 +0000 UTC m=+10.132912301 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e9240885-14d4-4b43-bf98-442e30d61c36-original-pull-secret") pod "global-pull-secret-syncer-s7zg6" (UID: "e9240885-14d4-4b43-bf98-442e30d61c36") : object "kube-system"/"original-pull-secret" not registered
Sep 16 17:24:53.762982 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:53.762946    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x46mt\" (UniqueName: \"kubernetes.io/projected/1d620b9d-0055-4ed6-aced-2f4cfc54cbd8-kube-api-access-x46mt\") pod \"network-check-target-4c79x\" (UID: \"1d620b9d-0055-4ed6-aced-2f4cfc54cbd8\") " pod="openshift-network-diagnostics/network-check-target-4c79x"
Sep 16 17:24:53.763195 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:24:53.763174    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 17:24:53.763279 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:24:53.763203    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 17:24:53.763279 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:24:53.763217    2579 projected.go:196] Error preparing data for projected volume kube-api-access-x46mt for pod openshift-network-diagnostics/network-check-target-4c79x: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 17:24:53.763279 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:24:53.763278    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1d620b9d-0055-4ed6-aced-2f4cfc54cbd8-kube-api-access-x46mt podName:1d620b9d-0055-4ed6-aced-2f4cfc54cbd8 nodeName:}" failed. No retries permitted until 2026-09-16 17:24:57.763259322 +0000 UTC m=+10.233257456 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-x46mt" (UniqueName: "kubernetes.io/projected/1d620b9d-0055-4ed6-aced-2f4cfc54cbd8-kube-api-access-x46mt") pod "network-check-target-4c79x" (UID: "1d620b9d-0055-4ed6-aced-2f4cfc54cbd8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 17:24:54.095039 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:54.094955    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4c79x"
Sep 16 17:24:54.095481 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:24:54.095083    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4c79x" podUID="1d620b9d-0055-4ed6-aced-2f4cfc54cbd8"
Sep 16 17:24:55.093229 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:55.093196    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xfgjq"
Sep 16 17:24:55.093413 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:24:55.093344    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xfgjq" podUID="bc719829-66b1-4a84-b994-e71cf111dafa"
Sep 16 17:24:55.093492 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:55.093416    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s7zg6"
Sep 16 17:24:55.093556 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:24:55.093531    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s7zg6" podUID="e9240885-14d4-4b43-bf98-442e30d61c36"
Sep 16 17:24:56.094657 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:56.094218    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4c79x"
Sep 16 17:24:56.094657 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:24:56.094353    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4c79x" podUID="1d620b9d-0055-4ed6-aced-2f4cfc54cbd8"
Sep 16 17:24:57.093446 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:57.093364    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xfgjq"
Sep 16 17:24:57.093606 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:57.093518    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s7zg6"
Sep 16 17:24:57.093727 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:24:57.093615    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s7zg6" podUID="e9240885-14d4-4b43-bf98-442e30d61c36"
Sep 16 17:24:57.093727 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:24:57.093514    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xfgjq" podUID="bc719829-66b1-4a84-b994-e71cf111dafa"
Sep 16 17:24:57.696860 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:57.696805    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e9240885-14d4-4b43-bf98-442e30d61c36-original-pull-secret\") pod \"global-pull-secret-syncer-s7zg6\" (UID: \"e9240885-14d4-4b43-bf98-442e30d61c36\") " pod="kube-system/global-pull-secret-syncer-s7zg6"
Sep 16 17:24:57.697310 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:57.696874    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bc719829-66b1-4a84-b994-e71cf111dafa-metrics-certs\") pod \"network-metrics-daemon-xfgjq\" (UID: \"bc719829-66b1-4a84-b994-e71cf111dafa\") " pod="openshift-multus/network-metrics-daemon-xfgjq"
Sep 16 17:24:57.697310 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:24:57.697007    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 17:24:57.697310 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:24:57.697061    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc719829-66b1-4a84-b994-e71cf111dafa-metrics-certs podName:bc719829-66b1-4a84-b994-e71cf111dafa nodeName:}" failed. No retries permitted until 2026-09-16 17:25:05.697046851 +0000 UTC m=+18.167044965 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bc719829-66b1-4a84-b994-e71cf111dafa-metrics-certs") pod "network-metrics-daemon-xfgjq" (UID: "bc719829-66b1-4a84-b994-e71cf111dafa") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 17:24:57.697310 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:24:57.697005    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 17:24:57.697310 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:24:57.697122    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e9240885-14d4-4b43-bf98-442e30d61c36-original-pull-secret podName:e9240885-14d4-4b43-bf98-442e30d61c36 nodeName:}" failed. No retries permitted until 2026-09-16 17:25:05.697108588 +0000 UTC m=+18.167106715 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e9240885-14d4-4b43-bf98-442e30d61c36-original-pull-secret") pod "global-pull-secret-syncer-s7zg6" (UID: "e9240885-14d4-4b43-bf98-442e30d61c36") : object "kube-system"/"original-pull-secret" not registered
Sep 16 17:24:57.798125 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:57.798084    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x46mt\" (UniqueName: \"kubernetes.io/projected/1d620b9d-0055-4ed6-aced-2f4cfc54cbd8-kube-api-access-x46mt\") pod \"network-check-target-4c79x\" (UID: \"1d620b9d-0055-4ed6-aced-2f4cfc54cbd8\") " pod="openshift-network-diagnostics/network-check-target-4c79x"
Sep 16 17:24:57.798361 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:24:57.798245    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 17:24:57.798361 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:24:57.798267    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 17:24:57.798361 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:24:57.798280    2579 projected.go:196] Error preparing data for projected volume kube-api-access-x46mt for pod openshift-network-diagnostics/network-check-target-4c79x: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 17:24:57.798361 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:24:57.798339    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1d620b9d-0055-4ed6-aced-2f4cfc54cbd8-kube-api-access-x46mt podName:1d620b9d-0055-4ed6-aced-2f4cfc54cbd8 nodeName:}" failed. No retries permitted until 2026-09-16 17:25:05.798321795 +0000 UTC m=+18.268319921 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-x46mt" (UniqueName: "kubernetes.io/projected/1d620b9d-0055-4ed6-aced-2f4cfc54cbd8-kube-api-access-x46mt") pod "network-check-target-4c79x" (UID: "1d620b9d-0055-4ed6-aced-2f4cfc54cbd8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 17:24:58.095420 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:58.095322    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4c79x"
Sep 16 17:24:58.095544 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:24:58.095433    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4c79x" podUID="1d620b9d-0055-4ed6-aced-2f4cfc54cbd8"
Sep 16 17:24:59.094143 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:59.094105    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xfgjq"
Sep 16 17:24:59.094592 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:24:59.094153    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s7zg6"
Sep 16 17:24:59.094592 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:24:59.094251    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xfgjq" podUID="bc719829-66b1-4a84-b994-e71cf111dafa"
Sep 16 17:24:59.094592 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:24:59.094390    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s7zg6" podUID="e9240885-14d4-4b43-bf98-442e30d61c36"
Sep 16 17:25:00.093362 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:25:00.093319    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4c79x"
Sep 16 17:25:00.093554 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:25:00.093450    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4c79x" podUID="1d620b9d-0055-4ed6-aced-2f4cfc54cbd8"
Sep 16 17:25:01.094083 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:25:01.094045    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xfgjq"
Sep 16 17:25:01.094083 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:25:01.094074    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s7zg6"
Sep 16 17:25:01.094547 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:25:01.094168    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xfgjq" podUID="bc719829-66b1-4a84-b994-e71cf111dafa"
Sep 16 17:25:01.094547 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:25:01.094302    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s7zg6" podUID="e9240885-14d4-4b43-bf98-442e30d61c36"
Sep 16 17:25:02.093927 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:25:02.093895    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4c79x"
Sep 16 17:25:02.094110 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:25:02.093997    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4c79x" podUID="1d620b9d-0055-4ed6-aced-2f4cfc54cbd8"
Sep 16 17:25:03.093353 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:25:03.093314    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xfgjq"
Sep 16 17:25:03.093548 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:25:03.093361    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s7zg6"
Sep 16 17:25:03.093548 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:25:03.093458    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xfgjq" podUID="bc719829-66b1-4a84-b994-e71cf111dafa"
Sep 16 17:25:03.093548 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:25:03.093544    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s7zg6" podUID="e9240885-14d4-4b43-bf98-442e30d61c36"
Sep 16 17:25:04.093180 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:25:04.093145    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4c79x"
Sep 16 17:25:04.093642 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:25:04.093263    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4c79x" podUID="1d620b9d-0055-4ed6-aced-2f4cfc54cbd8"
Sep 16 17:25:05.093098 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:25:05.093059    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xfgjq"
Sep 16 17:25:05.093325 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:25:05.093059    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s7zg6"
Sep 16 17:25:05.093325 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:25:05.093211    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xfgjq" podUID="bc719829-66b1-4a84-b994-e71cf111dafa"
Sep 16 17:25:05.093325 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:25:05.093318    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s7zg6" podUID="e9240885-14d4-4b43-bf98-442e30d61c36"
Sep 16 17:25:05.757046 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:25:05.757006    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e9240885-14d4-4b43-bf98-442e30d61c36-original-pull-secret\") pod \"global-pull-secret-syncer-s7zg6\" (UID: \"e9240885-14d4-4b43-bf98-442e30d61c36\") " pod="kube-system/global-pull-secret-syncer-s7zg6"
Sep 16 17:25:05.757248 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:25:05.757065    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bc719829-66b1-4a84-b994-e71cf111dafa-metrics-certs\") pod \"network-metrics-daemon-xfgjq\" (UID: \"bc719829-66b1-4a84-b994-e71cf111dafa\") " pod="openshift-multus/network-metrics-daemon-xfgjq"
Sep 16 17:25:05.757248 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:25:05.757138    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 17:25:05.757248 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:25:05.757177    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 17:25:05.757248 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:25:05.757217    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e9240885-14d4-4b43-bf98-442e30d61c36-original-pull-secret podName:e9240885-14d4-4b43-bf98-442e30d61c36 nodeName:}" failed. No retries permitted until 2026-09-16 17:25:21.757197573 +0000 UTC m=+34.227195695 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e9240885-14d4-4b43-bf98-442e30d61c36-original-pull-secret") pod "global-pull-secret-syncer-s7zg6" (UID: "e9240885-14d4-4b43-bf98-442e30d61c36") : object "kube-system"/"original-pull-secret" not registered
Sep 16 17:25:05.757248 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:25:05.757236    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc719829-66b1-4a84-b994-e71cf111dafa-metrics-certs podName:bc719829-66b1-4a84-b994-e71cf111dafa nodeName:}" failed. No retries permitted until 2026-09-16 17:25:21.75722775 +0000 UTC m=+34.227225862 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bc719829-66b1-4a84-b994-e71cf111dafa-metrics-certs") pod "network-metrics-daemon-xfgjq" (UID: "bc719829-66b1-4a84-b994-e71cf111dafa") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 17:25:05.857724 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:25:05.857689    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x46mt\" (UniqueName: \"kubernetes.io/projected/1d620b9d-0055-4ed6-aced-2f4cfc54cbd8-kube-api-access-x46mt\") pod \"network-check-target-4c79x\" (UID: \"1d620b9d-0055-4ed6-aced-2f4cfc54cbd8\") " pod="openshift-network-diagnostics/network-check-target-4c79x"
Sep 16 17:25:05.857921 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:25:05.857859    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 17:25:05.857921 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:25:05.857882    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 17:25:05.857921 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:25:05.857893    2579 projected.go:196] Error preparing data for projected volume kube-api-access-x46mt for pod openshift-network-diagnostics/network-check-target-4c79x: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 17:25:05.858042 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:25:05.857945    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1d620b9d-0055-4ed6-aced-2f4cfc54cbd8-kube-api-access-x46mt podName:1d620b9d-0055-4ed6-aced-2f4cfc54cbd8 nodeName:}" failed. No retries permitted until 2026-09-16 17:25:21.857929263 +0000 UTC m=+34.327927381 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-x46mt" (UniqueName: "kubernetes.io/projected/1d620b9d-0055-4ed6-aced-2f4cfc54cbd8-kube-api-access-x46mt") pod "network-check-target-4c79x" (UID: "1d620b9d-0055-4ed6-aced-2f4cfc54cbd8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 17:25:06.094163 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:25:06.094074    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4c79x"
Sep 16 17:25:06.094591 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:25:06.094208    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4c79x" podUID="1d620b9d-0055-4ed6-aced-2f4cfc54cbd8"
Sep 16 17:25:07.093177 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:25:07.093146    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xfgjq"
Sep 16 17:25:07.093371 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:25:07.093186    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s7zg6"
Sep 16 17:25:07.093371 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:25:07.093283    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xfgjq" podUID="bc719829-66b1-4a84-b994-e71cf111dafa"
Sep 16 17:25:07.093494 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:25:07.093390    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s7zg6" podUID="e9240885-14d4-4b43-bf98-442e30d61c36"
Sep 16 17:25:08.094091 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:25:08.094058    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4c79x"
Sep 16 17:25:08.094517 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:25:08.094145    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4c79x" podUID="1d620b9d-0055-4ed6-aced-2f4cfc54cbd8"
Sep 16 17:25:09.093516 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:25:09.093231    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s7zg6"
Sep 16 17:25:09.093660 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:25:09.093595    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s7zg6" podUID="e9240885-14d4-4b43-bf98-442e30d61c36"
Sep 16 17:25:09.093660 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:25:09.093274    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xfgjq"
Sep 16 17:25:09.093775 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:25:09.093712    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xfgjq" podUID="bc719829-66b1-4a84-b994-e71cf111dafa"
Sep 16 17:25:09.195162 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:25:09.195061    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-qtdvz" event={"ID":"21a2db47-94e4-4d2b-9cb2-0b2bca64ef5b","Type":"ContainerStarted","Data":"6a211b7686fefc32a534053906779e05bb46917f51e4c5ee1582d6a9b1347b0a"}
Sep 16 17:25:09.196634 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:25:09.196607    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-59rw5" event={"ID":"a1223407-0c9e-4ed6-bd17-cf0f3962e27e","Type":"ContainerStarted","Data":"2855bc22f9b6967a40a02881e365c18214a18aa34d414c58b91ef31ce7e06377"}
Sep 16 17:25:09.198952 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:25:09.198921    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-f9qxl" event={"ID":"c2617cf7-6369-4fc4-9fb5-6e6cfc4e9add","Type":"ContainerStarted","Data":"12fe23b1bf1b16102e32c5c97d8a3c65e5a2ab3f8ea0e064e8eaf83ea0a07298"}
Sep 16 17:25:09.199038 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:25:09.199009    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-f9qxl" event={"ID":"c2617cf7-6369-4fc4-9fb5-6e6cfc4e9add","Type":"ContainerStarted","Data":"2503ef11ee9bd2ebecd64bad271dd2d425adfbdc9bbe7d1e3e7c0847cfc50b69"}
Sep 16 17:25:09.199106 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:25:09.199059    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-f9qxl" event={"ID":"c2617cf7-6369-4fc4-9fb5-6e6cfc4e9add","Type":"ContainerStarted","Data":"253e8073d15fb71ccfbca5c6b86497ed8f2de15c24c10cce21b19dacc5468682"}
Sep 16 17:25:09.199151 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:25:09.199121    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-f9qxl" event={"ID":"c2617cf7-6369-4fc4-9fb5-6e6cfc4e9add","Type":"ContainerStarted","Data":"feab83e558b5de84586708d961e958d0e178565d8ea45e2420bfb60daa688e6d"}
Sep 16 17:25:09.200433 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:25:09.200406    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-gntnz" event={"ID":"ef27aa30-cb45-45bc-8d0e-50107fc6459d","Type":"ContainerStarted","Data":"c4c6c7527395b171ca45a57801d802077d674073f4725ed49cb215b159d280a0"}
Sep 16 17:25:09.202751 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:25:09.202726    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-pgq7f" event={"ID":"6bfc5b33-9432-43a1-a89a-8a2f18acc45f","Type":"ContainerStarted","Data":"296c177511c82351f362e44266375c947ddaed83063213804a4771b1678ac133"}
Sep 16 17:25:09.204287 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:25:09.204269    2579 generic.go:356] "Generic (PLEG): container finished" podID="5df1fb59-c460-463c-8691-2c270b69aa34" containerID="eca817d7fe9a1d638d2979c4fc3a3df486661ac53539d29cd6ab47987e144e47" exitCode=0
Sep 16 17:25:09.204353 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:25:09.204335    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6fsn7" event={"ID":"5df1fb59-c460-463c-8691-2c270b69aa34","Type":"ContainerDied","Data":"eca817d7fe9a1d638d2979c4fc3a3df486661ac53539d29cd6ab47987e144e47"}
Sep 16 17:25:09.205826 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:25:09.205809    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w58mp" event={"ID":"c73a8482-2abb-407e-972d-05a54828be7a","Type":"ContainerStarted","Data":"76146d12ba5c52da508724814f015c6e89faa7e078f7a004c83ae4e7ff5b9d47"}
Sep 16 17:25:09.207229 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:25:09.207209    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-t6cx6" event={"ID":"41b48733-fd0e-4eaa-a91c-5d57f642b2f8","Type":"ContainerStarted","Data":"13a6a42175b3d8bb533d971fde77b4bad250f4e0a0ff07b98134f49ec638228d"}
Sep 16 17:25:09.221446 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:25:09.221401    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-73.ec2.internal" podStartSLOduration=21.22138651 podStartE2EDuration="21.22138651s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 17:24:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:24:53.184151552 +0000 UTC m=+5.654149694" watchObservedRunningTime="2026-09-16 17:25:09.22138651 +0000 UTC m=+21.691384644"
Sep 16 17:25:09.221753 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:25:09.221734    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-qtdvz" podStartSLOduration=3.3277043969999998 podStartE2EDuration="21.221727267s" totalImagesPullingTime="17.89402287s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 17:24:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:25:09.221096446 +0000 UTC m=+21.691094581" watchObservedRunningTime="2026-09-16 17:25:09.221727267 +0000 UTC m=+21.691725403"
Sep 16 17:25:09.309621 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:25:09.309375    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-gntnz" podStartSLOduration=11.833323138 podStartE2EDuration="22.309354972s" totalImagesPullingTime="10.476031834s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:24:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:25:09.254748002 +0000 UTC m=+21.724746140" watchObservedRunningTime="2026-09-16 17:25:09.309354972 +0000 UTC m=+21.779353104"
Sep 16 17:25:09.310151 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:25:09.310095    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-pgq7f" podStartSLOduration=3.40383419 podStartE2EDuration="21.310080227s" totalImagesPullingTime="17.906246037s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 17:24:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:25:09.309677602 +0000 UTC m=+21.779675760" watchObservedRunningTime="2026-09-16 17:25:09.310080227 +0000 UTC m=+21.780078362"
Sep 16 17:25:09.359879 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:25:09.359816    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-59rw5" podStartSLOduration=4.344506443 podStartE2EDuration="22.359800835s" totalImagesPullingTime="18.015294392s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 17:24:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:25:09.359651743 +0000 UTC m=+21.829649877" watchObservedRunningTime="2026-09-16 17:25:09.359800835 +0000 UTC m=+21.829798970"
Sep 16 17:25:09.394477 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:25:09.394334    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-t6cx6" podStartSLOduration=3.510657622 podStartE2EDuration="21.394315313s" totalImagesPullingTime="17.883657691s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 17:24:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:25:09.393929143 +0000 UTC m=+21.863927279" watchObservedRunningTime="2026-09-16 17:25:09.394315313 +0000 UTC m=+21.864313448"
Sep 16 17:25:10.094122 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:25:10.094037    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4c79x"
Sep 16 17:25:10.094307 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:25:10.094171    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4c79x" podUID="1d620b9d-0055-4ed6-aced-2f4cfc54cbd8"
Sep 16 17:25:10.213990 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:25:10.213951    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-f9qxl" event={"ID":"c2617cf7-6369-4fc4-9fb5-6e6cfc4e9add","Type":"ContainerStarted","Data":"8a3b166b5dea4d1a2afab969630926be804994ca65dac9cad93b24f76371e9a5"}
Sep 16 17:25:10.214383 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:25:10.214048    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-f9qxl" event={"ID":"c2617cf7-6369-4fc4-9fb5-6e6cfc4e9add","Type":"ContainerStarted","Data":"5db2e484f6810a6f378aac496c2ac25d524be87a6f46a011852438c8a91e2f83"}
Sep 16 17:25:10.352290 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:25:10.352216    2579 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 16 17:25:10.981201 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:25:10.981065    2579 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-16T17:25:10.352243648Z","UUID":"177b0461-4194-45e5-8a98-6504388e968f","Handler":null,"Name":"","Endpoint":""}
Sep 16 17:25:10.982979 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:25:10.982952    2579 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 16 17:25:10.983121 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:25:10.982996    2579 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 16 17:25:11.093785 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:25:11.093752    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xfgjq"
Sep 16 17:25:11.093978 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:25:11.093752    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s7zg6"
Sep 16 17:25:11.093978 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:25:11.093920    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xfgjq" podUID="bc719829-66b1-4a84-b994-e71cf111dafa"
Sep 16 17:25:11.093978 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:25:11.093958    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s7zg6" podUID="e9240885-14d4-4b43-bf98-442e30d61c36"
Sep 16 17:25:11.219126 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:25:11.219091    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w58mp" event={"ID":"c73a8482-2abb-407e-972d-05a54828be7a","Type":"ContainerStarted","Data":"5415caf89606e5695452349d2087355130706da14085eed1d8b78bc80a44925d"}
Sep 16 17:25:11.221182 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:25:11.221147    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-gmmwp" event={"ID":"1ea5d21d-1046-4edb-b1d9-9ca43ebc564d","Type":"ContainerStarted","Data":"cd8a0b1f8faafa2c55c0170785081de5597138a6026ef5ad6ea0b057b60fb502"}
Sep 16 17:25:11.245920 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:25:11.245795    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-gmmwp" podStartSLOduration=6.342934844 podStartE2EDuration="24.245778937s" totalImagesPullingTime="17.902844093s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 17:24:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:25:11.245665515 +0000 UTC m=+23.715663676" watchObservedRunningTime="2026-09-16 17:25:11.245778937 +0000 UTC m=+23.715777071"
Sep 16 17:25:12.096698 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:25:12.096669    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4c79x"
Sep 16 17:25:12.096893 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:25:12.096770    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4c79x" podUID="1d620b9d-0055-4ed6-aced-2f4cfc54cbd8"
Sep 16 17:25:12.207254 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:25:12.207215    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-gntnz"
Sep 16 17:25:12.208028 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:25:12.207989    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-gntnz"
Sep 16 17:25:12.228079 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:25:12.228033    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-f9qxl" event={"ID":"c2617cf7-6369-4fc4-9fb5-6e6cfc4e9add","Type":"ContainerStarted","Data":"60aec5693974c345683e0273e940fc94eb183872a47f68043d38628d166a4783"}
Sep 16 17:25:12.230427 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:25:12.230397    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w58mp" event={"ID":"c73a8482-2abb-407e-972d-05a54828be7a","Type":"ContainerStarted","Data":"58c1b0e452638502f88bdfc96da9c39bf65264481898910a7aca368477ed7285"}
Sep 16 17:25:12.230622 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:25:12.230607    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-gntnz"
Sep 16 17:25:12.231139 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:25:12.231121    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-gntnz"
Sep 16 17:25:12.317914 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:25:12.317856    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w58mp" podStartSLOduration=5.09365141 podStartE2EDuration="25.317827584s" totalImagesPullingTime="20.224176174s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 17:24:47 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:25:12.290025131 +0000 UTC m=+24.760023267" watchObservedRunningTime="2026-09-16 17:25:12.317827584 +0000 UTC m=+24.787825718"
Sep 16 17:25:13.093617 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:25:13.093573    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xfgjq"
Sep 16 17:25:13.093813 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:25:13.093588    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s7zg6"
Sep 16 17:25:13.093813 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:25:13.093725    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xfgjq" podUID="bc719829-66b1-4a84-b994-e71cf111dafa"
Sep 16 17:25:13.093813 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:25:13.093793    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s7zg6" podUID="e9240885-14d4-4b43-bf98-442e30d61c36"
Sep 16 17:25:14.097508 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:25:14.097432    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4c79x"
Sep 16 17:25:14.098077 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:25:14.097561    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4c79x" podUID="1d620b9d-0055-4ed6-aced-2f4cfc54cbd8"
Sep 16 17:25:14.236613 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:25:14.236582    2579 generic.go:356] "Generic (PLEG): container finished" podID="5df1fb59-c460-463c-8691-2c270b69aa34" containerID="46b3f830eda62c65d16568cdcb353a2fccc1b639ccec108477deba0e883385df" exitCode=0
Sep 16 17:25:14.236765 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:25:14.236674    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6fsn7" event={"ID":"5df1fb59-c460-463c-8691-2c270b69aa34","Type":"ContainerDied","Data":"46b3f830eda62c65d16568cdcb353a2fccc1b639ccec108477deba0e883385df"}
Sep 16 17:25:14.240684 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:25:14.240656    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-f9qxl" event={"ID":"c2617cf7-6369-4fc4-9fb5-6e6cfc4e9add","Type":"ContainerStarted","Data":"53c8fc83f1b706959c0eb72485b9fb7ac271da30cdf5cf629a0d67f8880acb5a"}
Sep 16 17:25:14.241065 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:25:14.241040    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-f9qxl"
Sep 16 17:25:14.241190 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:25:14.241117    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-f9qxl"
Sep 16 17:25:14.256666 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:25:14.256639    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-f9qxl"
Sep 16 17:25:14.318245 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:25:14.318196    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-f9qxl" podStartSLOduration=8.397830464 podStartE2EDuration="26.31817947s" totalImagesPullingTime="17.920349006s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 17:24:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:25:14.318005333 +0000 UTC m=+26.788003467" watchObservedRunningTime="2026-09-16 17:25:14.31817947 +0000 UTC m=+26.788177605"
Sep 16 17:25:15.093366 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:25:15.093311    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xfgjq"
Sep 16 17:25:15.093366 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:25:15.093363    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s7zg6"
Sep 16 17:25:15.093570 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:25:15.093443    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xfgjq" podUID="bc719829-66b1-4a84-b994-e71cf111dafa"
Sep 16 17:25:15.093645 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:25:15.093573    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s7zg6" podUID="e9240885-14d4-4b43-bf98-442e30d61c36"
Sep 16 17:25:15.243506 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:25:15.243476    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-f9qxl"
Sep 16 17:25:15.262488 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:25:15.262459    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-f9qxl"
Sep 16 17:25:16.094201 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:25:16.094002    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4c79x"
Sep 16 17:25:16.094437 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:25:16.094286    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4c79x" podUID="1d620b9d-0055-4ed6-aced-2f4cfc54cbd8"
Sep 16 17:25:16.159366 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:25:16.159260    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-4c79x"]
Sep 16 17:25:16.162752 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:25:16.162715    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-s7zg6"]
Sep 16 17:25:16.162943 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:25:16.162862    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s7zg6"
Sep 16 17:25:16.163019 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:25:16.162989    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s7zg6" podUID="e9240885-14d4-4b43-bf98-442e30d61c36"
Sep 16 17:25:16.163403 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:25:16.163375    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-xfgjq"]
Sep 16 17:25:16.163512 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:25:16.163470    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xfgjq"
Sep 16 17:25:16.163570 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:25:16.163549    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xfgjq" podUID="bc719829-66b1-4a84-b994-e71cf111dafa"
Sep 16 17:25:16.247800 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:25:16.247769    2579 generic.go:356] "Generic (PLEG): container finished" podID="5df1fb59-c460-463c-8691-2c270b69aa34" containerID="2fc603c6edcec4ca7662e33caa3619aa1ab45dbcc134c2f3b05896cafb74dd72" exitCode=0
Sep 16 17:25:16.248284 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:25:16.247825    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6fsn7" event={"ID":"5df1fb59-c460-463c-8691-2c270b69aa34","Type":"ContainerDied","Data":"2fc603c6edcec4ca7662e33caa3619aa1ab45dbcc134c2f3b05896cafb74dd72"}
Sep 16 17:25:16.248284 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:25:16.247982    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4c79x"
Sep 16 17:25:16.248284 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:25:16.248091    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4c79x" podUID="1d620b9d-0055-4ed6-aced-2f4cfc54cbd8"
Sep 16 17:25:18.094824 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:25:18.094790    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s7zg6"
Sep 16 17:25:18.095259 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:25:18.094912    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4c79x"
Sep 16 17:25:18.095259 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:25:18.094916    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s7zg6" podUID="e9240885-14d4-4b43-bf98-442e30d61c36"
Sep 16 17:25:18.095259 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:25:18.094945    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xfgjq"
Sep 16 17:25:18.095259 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:25:18.095032    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4c79x" podUID="1d620b9d-0055-4ed6-aced-2f4cfc54cbd8"
Sep 16 17:25:18.095259 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:25:18.095106    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xfgjq" podUID="bc719829-66b1-4a84-b994-e71cf111dafa"
Sep 16 17:25:18.255233 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:25:18.255156    2579 generic.go:356] "Generic (PLEG): container finished" podID="5df1fb59-c460-463c-8691-2c270b69aa34" containerID="d33ab13014c1792877bd98cb10c46c249d3a83c8ff19aba4849332bbd49fa6fe" exitCode=0
Sep 16 17:25:18.255233 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:25:18.255214    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6fsn7" event={"ID":"5df1fb59-c460-463c-8691-2c270b69aa34","Type":"ContainerDied","Data":"d33ab13014c1792877bd98cb10c46c249d3a83c8ff19aba4849332bbd49fa6fe"}
Sep 16 17:25:20.093495 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:25:20.093286    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s7zg6"
Sep 16 17:25:20.093955 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:25:20.093291    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xfgjq"
Sep 16 17:25:20.093955 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:25:20.093606    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s7zg6" podUID="e9240885-14d4-4b43-bf98-442e30d61c36"
Sep 16 17:25:20.093955 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:25:20.093320    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4c79x"
Sep 16 17:25:20.093955 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:25:20.093685    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xfgjq" podUID="bc719829-66b1-4a84-b994-e71cf111dafa"
Sep 16 17:25:20.093955 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:25:20.093750    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4c79x" podUID="1d620b9d-0055-4ed6-aced-2f4cfc54cbd8"
Sep 16 17:25:21.776687 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:25:21.776644    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e9240885-14d4-4b43-bf98-442e30d61c36-original-pull-secret\") pod \"global-pull-secret-syncer-s7zg6\" (UID: \"e9240885-14d4-4b43-bf98-442e30d61c36\") " pod="kube-system/global-pull-secret-syncer-s7zg6"
Sep 16 17:25:21.776687 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:25:21.776699    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bc719829-66b1-4a84-b994-e71cf111dafa-metrics-certs\") pod \"network-metrics-daemon-xfgjq\" (UID: \"bc719829-66b1-4a84-b994-e71cf111dafa\") " pod="openshift-multus/network-metrics-daemon-xfgjq"
Sep 16 17:25:21.777283 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:25:21.776881    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 17:25:21.777283 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:25:21.776953    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc719829-66b1-4a84-b994-e71cf111dafa-metrics-certs podName:bc719829-66b1-4a84-b994-e71cf111dafa nodeName:}" failed. No retries permitted until 2026-09-16 17:25:53.776933503 +0000 UTC m=+66.246931621 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bc719829-66b1-4a84-b994-e71cf111dafa-metrics-certs") pod "network-metrics-daemon-xfgjq" (UID: "bc719829-66b1-4a84-b994-e71cf111dafa") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 17:25:21.777283 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:25:21.776879    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 17:25:21.777283 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:25:21.777046    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e9240885-14d4-4b43-bf98-442e30d61c36-original-pull-secret podName:e9240885-14d4-4b43-bf98-442e30d61c36 nodeName:}" failed. No retries permitted until 2026-09-16 17:25:53.777024643 +0000 UTC m=+66.247022761 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e9240885-14d4-4b43-bf98-442e30d61c36-original-pull-secret") pod "global-pull-secret-syncer-s7zg6" (UID: "e9240885-14d4-4b43-bf98-442e30d61c36") : object "kube-system"/"original-pull-secret" not registered
Sep 16 17:25:21.877340 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:25:21.877291    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x46mt\" (UniqueName: \"kubernetes.io/projected/1d620b9d-0055-4ed6-aced-2f4cfc54cbd8-kube-api-access-x46mt\") pod \"network-check-target-4c79x\" (UID: \"1d620b9d-0055-4ed6-aced-2f4cfc54cbd8\") " pod="openshift-network-diagnostics/network-check-target-4c79x"
Sep 16 17:25:21.877527 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:25:21.877488    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 17:25:21.877527 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:25:21.877514    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 17:25:21.877527 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:25:21.877528    2579 projected.go:196] Error preparing data for projected volume kube-api-access-x46mt for pod openshift-network-diagnostics/network-check-target-4c79x: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 17:25:21.877666 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:25:21.877587    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1d620b9d-0055-4ed6-aced-2f4cfc54cbd8-kube-api-access-x46mt podName:1d620b9d-0055-4ed6-aced-2f4cfc54cbd8 nodeName:}" failed. No retries permitted until 2026-09-16 17:25:53.877572351 +0000 UTC m=+66.347570478 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-x46mt" (UniqueName: "kubernetes.io/projected/1d620b9d-0055-4ed6-aced-2f4cfc54cbd8-kube-api-access-x46mt") pod "network-check-target-4c79x" (UID: "1d620b9d-0055-4ed6-aced-2f4cfc54cbd8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 17:25:22.094410 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:25:22.094190    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s7zg6"
Sep 16 17:25:22.094410 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:25:22.094207    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4c79x"
Sep 16 17:25:22.094410 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:25:22.094206    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xfgjq"
Sep 16 17:25:22.094410 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:25:22.094324    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s7zg6" podUID="e9240885-14d4-4b43-bf98-442e30d61c36"
Sep 16 17:25:22.094717 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:25:22.094467    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xfgjq" podUID="bc719829-66b1-4a84-b994-e71cf111dafa"
Sep 16 17:25:22.094717 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:25:22.094529    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4c79x" podUID="1d620b9d-0055-4ed6-aced-2f4cfc54cbd8"
Sep 16 17:25:22.760229 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:25:22.760188    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-73.ec2.internal" event="NodeReady"
Sep 16 17:25:22.760432 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:25:22.760382    2579 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 16 17:25:22.855067 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:25:22.855025    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-9cbqd"]
Sep 16 17:25:22.857645 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:25:22.857623    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-t8qph"]
Sep 16 17:25:22.857790 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:25:22.857755    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-9cbqd"
Sep 16 17:25:22.859592 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:25:22.859573    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-t8qph"
Sep 16 17:25:22.864070 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:25:22.863678    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 16 17:25:22.864070 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:25:22.863876    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-v8tmk\""
Sep 16 17:25:22.864070 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:25:22.863891    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 16 17:25:22.864070 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:25:22.863925    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 16 17:25:22.864070 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:25:22.863948    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 16 17:25:22.864388 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:25:22.864213    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 16 17:25:22.864477 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:25:22.864458    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-nqgn2\""
Sep 16 17:25:22.871674 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:25:22.871653    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-t8qph"]
Sep 16 17:25:22.882807 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:25:22.882777    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-9cbqd"]
Sep 16 17:25:22.986148 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:25:22.986121    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a639ab67-695c-4ec2-af20-1882ac49c4df-metrics-tls\") pod \"dns-default-9cbqd\" (UID: \"a639ab67-695c-4ec2-af20-1882ac49c4df\") " pod="openshift-dns/dns-default-9cbqd"
Sep 16 17:25:22.986307 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:25:22.986158    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4kmsl\" (UniqueName: \"kubernetes.io/projected/7886f4b3-cb86-4a0e-ae6c-809ac7a567fa-kube-api-access-4kmsl\") pod \"ingress-canary-t8qph\" (UID: \"7886f4b3-cb86-4a0e-ae6c-809ac7a567fa\") " pod="openshift-ingress-canary/ingress-canary-t8qph"
Sep 16 17:25:22.986307 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:25:22.986180    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a639ab67-695c-4ec2-af20-1882ac49c4df-config-volume\") pod \"dns-default-9cbqd\" (UID: \"a639ab67-695c-4ec2-af20-1882ac49c4df\") " pod="openshift-dns/dns-default-9cbqd"
Sep 16 17:25:22.986307 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:25:22.986295    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a639ab67-695c-4ec2-af20-1882ac49c4df-tmp-dir\") pod \"dns-default-9cbqd\" (UID: \"a639ab67-695c-4ec2-af20-1882ac49c4df\") " pod="openshift-dns/dns-default-9cbqd"
Sep 16 17:25:22.986474 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:25:22.986410    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvzx8\" (UniqueName: \"kubernetes.io/projected/a639ab67-695c-4ec2-af20-1882ac49c4df-kube-api-access-dvzx8\") pod \"dns-default-9cbqd\" (UID: \"a639ab67-695c-4ec2-af20-1882ac49c4df\") " pod="openshift-dns/dns-default-9cbqd"
Sep 16 17:25:22.986519 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:25:22.986484    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7886f4b3-cb86-4a0e-ae6c-809ac7a567fa-cert\") pod \"ingress-canary-t8qph\" (UID: \"7886f4b3-cb86-4a0e-ae6c-809ac7a567fa\") " pod="openshift-ingress-canary/ingress-canary-t8qph"
Sep 16 17:25:23.087163 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:25:23.087127    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a639ab67-695c-4ec2-af20-1882ac49c4df-tmp-dir\") pod \"dns-default-9cbqd\" (UID: \"a639ab67-695c-4ec2-af20-1882ac49c4df\") " pod="openshift-dns/dns-default-9cbqd"
Sep 16 17:25:23.087372 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:25:23.087185    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvzx8\" (UniqueName: \"kubernetes.io/projected/a639ab67-695c-4ec2-af20-1882ac49c4df-kube-api-access-dvzx8\") pod \"dns-default-9cbqd\" (UID: \"a639ab67-695c-4ec2-af20-1882ac49c4df\") " pod="openshift-dns/dns-default-9cbqd"
Sep 16 17:25:23.087372 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:25:23.087206    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7886f4b3-cb86-4a0e-ae6c-809ac7a567fa-cert\") pod \"ingress-canary-t8qph\" (UID: \"7886f4b3-cb86-4a0e-ae6c-809ac7a567fa\") " pod="openshift-ingress-canary/ingress-canary-t8qph"
Sep 16 17:25:23.087372 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:25:23.087235    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a639ab67-695c-4ec2-af20-1882ac49c4df-metrics-tls\") pod \"dns-default-9cbqd\" (UID: \"a639ab67-695c-4ec2-af20-1882ac49c4df\") " pod="openshift-dns/dns-default-9cbqd"
Sep 16 17:25:23.087372 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:25:23.087251    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4kmsl\" (UniqueName: \"kubernetes.io/projected/7886f4b3-cb86-4a0e-ae6c-809ac7a567fa-kube-api-access-4kmsl\") pod \"ingress-canary-t8qph\" (UID: \"7886f4b3-cb86-4a0e-ae6c-809ac7a567fa\") " pod="openshift-ingress-canary/ingress-canary-t8qph"
Sep 16 17:25:23.087372 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:25:23.087268    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a639ab67-695c-4ec2-af20-1882ac49c4df-config-volume\") pod \"dns-default-9cbqd\" (UID: \"a639ab67-695c-4ec2-af20-1882ac49c4df\") " pod="openshift-dns/dns-default-9cbqd"
Sep 16 17:25:23.087680 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:25:23.087388    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 17:25:23.087680 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:25:23.087455    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a639ab67-695c-4ec2-af20-1882ac49c4df-metrics-tls podName:a639ab67-695c-4ec2-af20-1882ac49c4df nodeName:}" failed. No retries permitted until 2026-09-16 17:25:23.587436327 +0000 UTC m=+36.057434458 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a639ab67-695c-4ec2-af20-1882ac49c4df-metrics-tls") pod "dns-default-9cbqd" (UID: "a639ab67-695c-4ec2-af20-1882ac49c4df") : secret "dns-default-metrics-tls" not found
Sep 16 17:25:23.087680 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:25:23.087505    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a639ab67-695c-4ec2-af20-1882ac49c4df-tmp-dir\") pod \"dns-default-9cbqd\" (UID: \"a639ab67-695c-4ec2-af20-1882ac49c4df\") " pod="openshift-dns/dns-default-9cbqd"
Sep 16 17:25:23.087898 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:25:23.087879    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a639ab67-695c-4ec2-af20-1882ac49c4df-config-volume\") pod \"dns-default-9cbqd\" (UID: \"a639ab67-695c-4ec2-af20-1882ac49c4df\") " pod="openshift-dns/dns-default-9cbqd"
Sep 16 17:25:23.091450 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:25:23.091423    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7886f4b3-cb86-4a0e-ae6c-809ac7a567fa-cert\") pod \"ingress-canary-t8qph\" (UID: \"7886f4b3-cb86-4a0e-ae6c-809ac7a567fa\") " pod="openshift-ingress-canary/ingress-canary-t8qph"
Sep 16 17:25:23.105673 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:25:23.105648    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvzx8\" (UniqueName: \"kubernetes.io/projected/a639ab67-695c-4ec2-af20-1882ac49c4df-kube-api-access-dvzx8\") pod \"dns-default-9cbqd\" (UID: \"a639ab67-695c-4ec2-af20-1882ac49c4df\") " pod="openshift-dns/dns-default-9cbqd"
Sep 16 17:25:23.106082 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:25:23.106060    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4kmsl\" (UniqueName: \"kubernetes.io/projected/7886f4b3-cb86-4a0e-ae6c-809ac7a567fa-kube-api-access-4kmsl\") pod \"ingress-canary-t8qph\" (UID: \"7886f4b3-cb86-4a0e-ae6c-809ac7a567fa\") " pod="openshift-ingress-canary/ingress-canary-t8qph"
Sep 16 17:25:23.176976 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:25:23.176944    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-t8qph"
Sep 16 17:25:23.268618 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:25:23.268533    2579 generic.go:356] "Generic (PLEG): container finished" podID="5df1fb59-c460-463c-8691-2c270b69aa34" containerID="aeb4d64f556c6545ce9ef0d981884ef42bd171efd4549d14b79d982f2c9e6525" exitCode=0
Sep 16 17:25:23.268618 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:25:23.268583    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6fsn7" event={"ID":"5df1fb59-c460-463c-8691-2c270b69aa34","Type":"ContainerDied","Data":"aeb4d64f556c6545ce9ef0d981884ef42bd171efd4549d14b79d982f2c9e6525"}
Sep 16 17:25:23.462722 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:25:23.462468    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-t8qph"]
Sep 16 17:25:23.466197 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:25:23.466171    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7886f4b3_cb86_4a0e_ae6c_809ac7a567fa.slice/crio-d12e16344dd85b5ac06b10e436e38b48eb313fc87477532ca49db73f7cf0e0e8 WatchSource:0}: Error finding container d12e16344dd85b5ac06b10e436e38b48eb313fc87477532ca49db73f7cf0e0e8: Status 404 returned error can't find the container with id d12e16344dd85b5ac06b10e436e38b48eb313fc87477532ca49db73f7cf0e0e8
Sep 16 17:25:23.591136 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:25:23.591103    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a639ab67-695c-4ec2-af20-1882ac49c4df-metrics-tls\") pod \"dns-default-9cbqd\" (UID: \"a639ab67-695c-4ec2-af20-1882ac49c4df\") " pod="openshift-dns/dns-default-9cbqd"
Sep 16 17:25:23.591287 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:25:23.591239    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 17:25:23.591325 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:25:23.591304    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a639ab67-695c-4ec2-af20-1882ac49c4df-metrics-tls podName:a639ab67-695c-4ec2-af20-1882ac49c4df nodeName:}" failed. No retries permitted until 2026-09-16 17:25:24.59128979 +0000 UTC m=+37.061287908 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a639ab67-695c-4ec2-af20-1882ac49c4df-metrics-tls") pod "dns-default-9cbqd" (UID: "a639ab67-695c-4ec2-af20-1882ac49c4df") : secret "dns-default-metrics-tls" not found
Sep 16 17:25:24.098207 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:25:24.098175    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4c79x"
Sep 16 17:25:24.098207 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:25:24.098195    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s7zg6"
Sep 16 17:25:24.099023 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:25:24.098378    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xfgjq"
Sep 16 17:25:24.106539 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:25:24.106066    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 16 17:25:24.106539 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:25:24.106236    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 16 17:25:24.107241 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:25:24.107204    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 16 17:25:24.107241 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:25:24.107220    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 16 17:25:24.107426 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:25:24.107254    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-2lh24\""
Sep 16 17:25:24.107426 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:25:24.107214    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-q452j\""
Sep 16 17:25:24.272610 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:25:24.272574    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-t8qph" event={"ID":"7886f4b3-cb86-4a0e-ae6c-809ac7a567fa","Type":"ContainerStarted","Data":"d12e16344dd85b5ac06b10e436e38b48eb313fc87477532ca49db73f7cf0e0e8"}
Sep 16 17:25:24.276137 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:25:24.276108    2579 generic.go:356] "Generic (PLEG): container finished" podID="5df1fb59-c460-463c-8691-2c270b69aa34" containerID="09b4adaecd463b978edd47328bafa5b400f6763bffff11aabf2535ad42ac5fdc" exitCode=0
Sep 16 17:25:24.276276 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:25:24.276161    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6fsn7" event={"ID":"5df1fb59-c460-463c-8691-2c270b69aa34","Type":"ContainerDied","Data":"09b4adaecd463b978edd47328bafa5b400f6763bffff11aabf2535ad42ac5fdc"}
Sep 16 17:25:24.598590 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:25:24.598557    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a639ab67-695c-4ec2-af20-1882ac49c4df-metrics-tls\") pod \"dns-default-9cbqd\" (UID: \"a639ab67-695c-4ec2-af20-1882ac49c4df\") " pod="openshift-dns/dns-default-9cbqd"
Sep 16 17:25:24.598747 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:25:24.598710    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 17:25:24.598817 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:25:24.598788    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a639ab67-695c-4ec2-af20-1882ac49c4df-metrics-tls podName:a639ab67-695c-4ec2-af20-1882ac49c4df nodeName:}" failed. No retries permitted until 2026-09-16 17:25:26.59877109 +0000 UTC m=+39.068769227 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a639ab67-695c-4ec2-af20-1882ac49c4df-metrics-tls") pod "dns-default-9cbqd" (UID: "a639ab67-695c-4ec2-af20-1882ac49c4df") : secret "dns-default-metrics-tls" not found
Sep 16 17:25:25.283347 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:25:25.283309    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6fsn7" event={"ID":"5df1fb59-c460-463c-8691-2c270b69aa34","Type":"ContainerStarted","Data":"30a19ab88fcdbe62b8e24766d7f03b1e7eecb54684aede212c0e1ae3943e941d"}
Sep 16 17:25:26.287514 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:25:26.287473    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-t8qph" event={"ID":"7886f4b3-cb86-4a0e-ae6c-809ac7a567fa","Type":"ContainerStarted","Data":"049e15d60244d6e6ea0fdc40b259e15f47fa153b29069d392c3bea897aa1f80e"}
Sep 16 17:25:26.319827 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:25:26.319757    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-t8qph" podStartSLOduration=1.9065152570000001 podStartE2EDuration="4.319740833s" totalImagesPullingTime="2.413225576s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:25:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:25:26.31955814 +0000 UTC m=+38.789556277" watchObservedRunningTime="2026-09-16 17:25:26.319740833 +0000 UTC m=+38.789738967"
Sep 16 17:25:26.320059 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:25:26.320040    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-6fsn7" podStartSLOduration=9.125392268 podStartE2EDuration="38.320033748s" totalImagesPullingTime="29.19464148s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 17:24:48 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:25:25.314285339 +0000 UTC m=+37.784283475" watchObservedRunningTime="2026-09-16 17:25:26.320033748 +0000 UTC m=+38.790031883"
Sep 16 17:25:26.615120 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:25:26.615026    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a639ab67-695c-4ec2-af20-1882ac49c4df-metrics-tls\") pod \"dns-default-9cbqd\" (UID: \"a639ab67-695c-4ec2-af20-1882ac49c4df\") " pod="openshift-dns/dns-default-9cbqd"
Sep 16 17:25:26.615251 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:25:26.615182    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 17:25:26.615292 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:25:26.615250    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a639ab67-695c-4ec2-af20-1882ac49c4df-metrics-tls podName:a639ab67-695c-4ec2-af20-1882ac49c4df nodeName:}" failed. No retries permitted until 2026-09-16 17:25:30.61523226 +0000 UTC m=+43.085230373 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a639ab67-695c-4ec2-af20-1882ac49c4df-metrics-tls") pod "dns-default-9cbqd" (UID: "a639ab67-695c-4ec2-af20-1882ac49c4df") : secret "dns-default-metrics-tls" not found
Sep 16 17:25:30.638963 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:25:30.638924    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a639ab67-695c-4ec2-af20-1882ac49c4df-metrics-tls\") pod \"dns-default-9cbqd\" (UID: \"a639ab67-695c-4ec2-af20-1882ac49c4df\") " pod="openshift-dns/dns-default-9cbqd"
Sep 16 17:25:30.639335 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:25:30.639033    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 17:25:30.639335 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:25:30.639097    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a639ab67-695c-4ec2-af20-1882ac49c4df-metrics-tls podName:a639ab67-695c-4ec2-af20-1882ac49c4df nodeName:}" failed. No retries permitted until 2026-09-16 17:25:38.639082399 +0000 UTC m=+51.109080512 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a639ab67-695c-4ec2-af20-1882ac49c4df-metrics-tls") pod "dns-default-9cbqd" (UID: "a639ab67-695c-4ec2-af20-1882ac49c4df") : secret "dns-default-metrics-tls" not found
Sep 16 17:25:38.695218 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:25:38.695179    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a639ab67-695c-4ec2-af20-1882ac49c4df-metrics-tls\") pod \"dns-default-9cbqd\" (UID: \"a639ab67-695c-4ec2-af20-1882ac49c4df\") " pod="openshift-dns/dns-default-9cbqd"
Sep 16 17:25:38.695658 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:25:38.695297    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 17:25:38.695658 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:25:38.695363    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a639ab67-695c-4ec2-af20-1882ac49c4df-metrics-tls podName:a639ab67-695c-4ec2-af20-1882ac49c4df nodeName:}" failed. No retries permitted until 2026-09-16 17:25:54.695348769 +0000 UTC m=+67.165346882 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a639ab67-695c-4ec2-af20-1882ac49c4df-metrics-tls") pod "dns-default-9cbqd" (UID: "a639ab67-695c-4ec2-af20-1882ac49c4df") : secret "dns-default-metrics-tls" not found
Sep 16 17:25:47.270650 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:25:47.270619    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-f9qxl"
Sep 16 17:25:53.803949 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:25:53.803898    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e9240885-14d4-4b43-bf98-442e30d61c36-original-pull-secret\") pod \"global-pull-secret-syncer-s7zg6\" (UID: \"e9240885-14d4-4b43-bf98-442e30d61c36\") " pod="kube-system/global-pull-secret-syncer-s7zg6"
Sep 16 17:25:53.803949 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:25:53.803946    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bc719829-66b1-4a84-b994-e71cf111dafa-metrics-certs\") pod \"network-metrics-daemon-xfgjq\" (UID: \"bc719829-66b1-4a84-b994-e71cf111dafa\") " pod="openshift-multus/network-metrics-daemon-xfgjq"
Sep 16 17:25:53.806575 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:25:53.806552    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 16 17:25:53.806638 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:25:53.806594    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 16 17:25:53.814711 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:25:53.814694    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 16 17:25:53.814806 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:25:53.814796    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc719829-66b1-4a84-b994-e71cf111dafa-metrics-certs podName:bc719829-66b1-4a84-b994-e71cf111dafa nodeName:}" failed. No retries permitted until 2026-09-16 17:26:57.814780327 +0000 UTC m=+130.284778440 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bc719829-66b1-4a84-b994-e71cf111dafa-metrics-certs") pod "network-metrics-daemon-xfgjq" (UID: "bc719829-66b1-4a84-b994-e71cf111dafa") : secret "metrics-daemon-secret" not found
Sep 16 17:25:53.818075 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:25:53.818049    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e9240885-14d4-4b43-bf98-442e30d61c36-original-pull-secret\") pod \"global-pull-secret-syncer-s7zg6\" (UID: \"e9240885-14d4-4b43-bf98-442e30d61c36\") " pod="kube-system/global-pull-secret-syncer-s7zg6"
Sep 16 17:25:53.904752 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:25:53.904715    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x46mt\" (UniqueName: \"kubernetes.io/projected/1d620b9d-0055-4ed6-aced-2f4cfc54cbd8-kube-api-access-x46mt\") pod \"network-check-target-4c79x\" (UID: \"1d620b9d-0055-4ed6-aced-2f4cfc54cbd8\") " pod="openshift-network-diagnostics/network-check-target-4c79x"
Sep 16 17:25:53.907628 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:25:53.907611    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 16 17:25:53.917562 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:25:53.917537    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 16 17:25:53.928893 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:25:53.928866    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x46mt\" (UniqueName: \"kubernetes.io/projected/1d620b9d-0055-4ed6-aced-2f4cfc54cbd8-kube-api-access-x46mt\") pod \"network-check-target-4c79x\" (UID: \"1d620b9d-0055-4ed6-aced-2f4cfc54cbd8\") " pod="openshift-network-diagnostics/network-check-target-4c79x"
Sep 16 17:25:54.112767 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:25:54.112687    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-2lh24\""
Sep 16 17:25:54.116537 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:25:54.116521    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s7zg6"
Sep 16 17:25:54.120660 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:25:54.120644    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4c79x"
Sep 16 17:25:54.412359 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:25:54.412277    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-s7zg6"]
Sep 16 17:25:54.415585 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:25:54.415560    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9240885_14d4_4b43_bf98_442e30d61c36.slice/crio-7fa6ddbf64f0a32ac6b054c30a8226aaec4b462eba90e226e33f3a5b548c7794 WatchSource:0}: Error finding container 7fa6ddbf64f0a32ac6b054c30a8226aaec4b462eba90e226e33f3a5b548c7794: Status 404 returned error can't find the container with id 7fa6ddbf64f0a32ac6b054c30a8226aaec4b462eba90e226e33f3a5b548c7794
Sep 16 17:25:54.437753 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:25:54.437713    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-4c79x"]
Sep 16 17:25:54.442761 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:25:54.442737    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d620b9d_0055_4ed6_aced_2f4cfc54cbd8.slice/crio-ab936a0e367a19f8e7d1285a093a30ad8e01129221f6dfdabd8c12b0bcaffcd6 WatchSource:0}: Error finding container ab936a0e367a19f8e7d1285a093a30ad8e01129221f6dfdabd8c12b0bcaffcd6: Status 404 returned error can't find the container with id ab936a0e367a19f8e7d1285a093a30ad8e01129221f6dfdabd8c12b0bcaffcd6
Sep 16 17:25:54.711042 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:25:54.710950    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a639ab67-695c-4ec2-af20-1882ac49c4df-metrics-tls\") pod \"dns-default-9cbqd\" (UID: \"a639ab67-695c-4ec2-af20-1882ac49c4df\") " pod="openshift-dns/dns-default-9cbqd"
Sep 16 17:25:54.711189 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:25:54.711078    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 17:25:54.711189 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:25:54.711146    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a639ab67-695c-4ec2-af20-1882ac49c4df-metrics-tls podName:a639ab67-695c-4ec2-af20-1882ac49c4df nodeName:}" failed. No retries permitted until 2026-09-16 17:26:26.711132212 +0000 UTC m=+99.181130324 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a639ab67-695c-4ec2-af20-1882ac49c4df-metrics-tls") pod "dns-default-9cbqd" (UID: "a639ab67-695c-4ec2-af20-1882ac49c4df") : secret "dns-default-metrics-tls" not found
Sep 16 17:25:55.359401 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:25:55.359331    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-4c79x" event={"ID":"1d620b9d-0055-4ed6-aced-2f4cfc54cbd8","Type":"ContainerStarted","Data":"ab936a0e367a19f8e7d1285a093a30ad8e01129221f6dfdabd8c12b0bcaffcd6"}
Sep 16 17:25:55.360730 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:25:55.360699    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-s7zg6" event={"ID":"e9240885-14d4-4b43-bf98-442e30d61c36","Type":"ContainerStarted","Data":"7fa6ddbf64f0a32ac6b054c30a8226aaec4b462eba90e226e33f3a5b548c7794"}
Sep 16 17:25:59.373239 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:25:59.373200    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-s7zg6" event={"ID":"e9240885-14d4-4b43-bf98-442e30d61c36","Type":"ContainerStarted","Data":"21c1597a08c5aa3fd0057ecb0c0524fb3b64468cd1b5be08a1818dd6626c520f"}
Sep 16 17:25:59.374624 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:25:59.374600    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-4c79x" event={"ID":"1d620b9d-0055-4ed6-aced-2f4cfc54cbd8","Type":"ContainerStarted","Data":"52fcb5df47410cac766a3c3ddf94ee3456ce2187325d1b2a89aa67034dee4d7a"}
Sep 16 17:25:59.374740 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:25:59.374728    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-4c79x"
Sep 16 17:25:59.397079 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:25:59.397034    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-s7zg6" podStartSLOduration=67.102864387 podStartE2EDuration="1m11.397020305s" totalImagesPullingTime="4.294155918s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 17:24:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:25:59.396480618 +0000 UTC m=+71.866478754" watchObservedRunningTime="2026-09-16 17:25:59.397020305 +0000 UTC m=+71.867018441"
Sep 16 17:26:26.723988 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:26:26.723929    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a639ab67-695c-4ec2-af20-1882ac49c4df-metrics-tls\") pod \"dns-default-9cbqd\" (UID: \"a639ab67-695c-4ec2-af20-1882ac49c4df\") " pod="openshift-dns/dns-default-9cbqd"
Sep 16 17:26:26.724430 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:26:26.724105    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 17:26:26.724430 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:26:26.724171    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a639ab67-695c-4ec2-af20-1882ac49c4df-metrics-tls podName:a639ab67-695c-4ec2-af20-1882ac49c4df nodeName:}" failed. No retries permitted until 2026-09-16 17:27:30.724152723 +0000 UTC m=+163.194150841 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a639ab67-695c-4ec2-af20-1882ac49c4df-metrics-tls") pod "dns-default-9cbqd" (UID: "a639ab67-695c-4ec2-af20-1882ac49c4df") : secret "dns-default-metrics-tls" not found
Sep 16 17:26:30.379190 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:26:30.379158    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-4c79x"
Sep 16 17:26:30.395005 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:26:30.394956    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-4c79x" podStartSLOduration=98.132027852 podStartE2EDuration="1m42.394942006s" totalImagesPullingTime="4.262914154s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:24:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:25:59.41657469 +0000 UTC m=+71.886572836" watchObservedRunningTime="2026-09-16 17:26:30.394942006 +0000 UTC m=+102.864940140"
Sep 16 17:26:48.528265 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:26:48.528230    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-99d9f4-g4nhp"]
Sep 16 17:26:48.530113 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:26:48.530095    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-99d9f4-g4nhp"
Sep 16 17:26:48.532768 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:26:48.532746    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Sep 16 17:26:48.532940 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:26:48.532748    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Sep 16 17:26:48.533312 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:26:48.533297    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Sep 16 17:26:48.533382 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:26:48.533366    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Sep 16 17:26:48.533467 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:26:48.533437    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 16 17:26:48.534405 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:26:48.534386    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-kbl7s\""
Sep 16 17:26:48.534475 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:26:48.534440    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 16 17:26:48.548782 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:26:48.548760    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-99d9f4-g4nhp"]
Sep 16 17:26:48.568239 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:26:48.568205    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7a9b26b4-3f5f-40d2-b866-ad3915e771bb-service-ca-bundle\") pod \"router-default-99d9f4-g4nhp\" (UID: \"7a9b26b4-3f5f-40d2-b866-ad3915e771bb\") " pod="openshift-ingress/router-default-99d9f4-g4nhp"
Sep 16 17:26:48.568239 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:26:48.568243    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6jsq\" (UniqueName: \"kubernetes.io/projected/7a9b26b4-3f5f-40d2-b866-ad3915e771bb-kube-api-access-k6jsq\") pod \"router-default-99d9f4-g4nhp\" (UID: \"7a9b26b4-3f5f-40d2-b866-ad3915e771bb\") " pod="openshift-ingress/router-default-99d9f4-g4nhp"
Sep 16 17:26:48.568426 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:26:48.568262    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/7a9b26b4-3f5f-40d2-b866-ad3915e771bb-default-certificate\") pod \"router-default-99d9f4-g4nhp\" (UID: \"7a9b26b4-3f5f-40d2-b866-ad3915e771bb\") " pod="openshift-ingress/router-default-99d9f4-g4nhp"
Sep 16 17:26:48.568426 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:26:48.568313    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/7a9b26b4-3f5f-40d2-b866-ad3915e771bb-stats-auth\") pod \"router-default-99d9f4-g4nhp\" (UID: \"7a9b26b4-3f5f-40d2-b866-ad3915e771bb\") " pod="openshift-ingress/router-default-99d9f4-g4nhp"
Sep 16 17:26:48.568426 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:26:48.568351    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a9b26b4-3f5f-40d2-b866-ad3915e771bb-metrics-certs\") pod \"router-default-99d9f4-g4nhp\" (UID: \"7a9b26b4-3f5f-40d2-b866-ad3915e771bb\") " pod="openshift-ingress/router-default-99d9f4-g4nhp"
Sep 16 17:26:48.634962 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:26:48.634933    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-f7kcw"]
Sep 16 17:26:48.636707 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:26:48.636690    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-f7kcw"
Sep 16 17:26:48.643833 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:26:48.643808    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Sep 16 17:26:48.643997 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:26:48.643956    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-ft6rf\""
Sep 16 17:26:48.644053 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:26:48.643957    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Sep 16 17:26:48.644103 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:26:48.644024    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Sep 16 17:26:48.644288 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:26:48.644273    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Sep 16 17:26:48.653358 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:26:48.653331    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-f7kcw"]
Sep 16 17:26:48.669602 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:26:48.669572    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7a9b26b4-3f5f-40d2-b866-ad3915e771bb-service-ca-bundle\") pod \"router-default-99d9f4-g4nhp\" (UID: \"7a9b26b4-3f5f-40d2-b866-ad3915e771bb\") " pod="openshift-ingress/router-default-99d9f4-g4nhp"
Sep 16 17:26:48.669602 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:26:48.669605    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6jsq\" (UniqueName: \"kubernetes.io/projected/7a9b26b4-3f5f-40d2-b866-ad3915e771bb-kube-api-access-k6jsq\") pod \"router-default-99d9f4-g4nhp\" (UID: \"7a9b26b4-3f5f-40d2-b866-ad3915e771bb\") " pod="openshift-ingress/router-default-99d9f4-g4nhp"
Sep 16 17:26:48.669801 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:26:48.669624    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/7a9b26b4-3f5f-40d2-b866-ad3915e771bb-default-certificate\") pod \"router-default-99d9f4-g4nhp\" (UID: \"7a9b26b4-3f5f-40d2-b866-ad3915e771bb\") " pod="openshift-ingress/router-default-99d9f4-g4nhp"
Sep 16 17:26:48.669801 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:26:48.669646    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/51b86650-5910-4bc0-9832-dc967737cbea-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-f7kcw\" (UID: \"51b86650-5910-4bc0-9832-dc967737cbea\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-f7kcw"
Sep 16 17:26:48.669801 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:26:48.669676    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/7a9b26b4-3f5f-40d2-b866-ad3915e771bb-stats-auth\") pod \"router-default-99d9f4-g4nhp\" (UID: \"7a9b26b4-3f5f-40d2-b866-ad3915e771bb\") " pod="openshift-ingress/router-default-99d9f4-g4nhp"
Sep 16 17:26:48.669801 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:26:48.669698    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/51b86650-5910-4bc0-9832-dc967737cbea-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-f7kcw\" (UID: \"51b86650-5910-4bc0-9832-dc967737cbea\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-f7kcw"
Sep 16 17:26:48.669801 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:26:48.669725    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smfch\" (UniqueName: \"kubernetes.io/projected/51b86650-5910-4bc0-9832-dc967737cbea-kube-api-access-smfch\") pod \"kube-storage-version-migrator-operator-6c8f84f897-f7kcw\" (UID: \"51b86650-5910-4bc0-9832-dc967737cbea\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-f7kcw"
Sep 16 17:26:48.669801 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:26:48.669759    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a9b26b4-3f5f-40d2-b866-ad3915e771bb-metrics-certs\") pod \"router-default-99d9f4-g4nhp\" (UID: \"7a9b26b4-3f5f-40d2-b866-ad3915e771bb\") " pod="openshift-ingress/router-default-99d9f4-g4nhp"
Sep 16 17:26:48.669801 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:26:48.669792    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7a9b26b4-3f5f-40d2-b866-ad3915e771bb-service-ca-bundle podName:7a9b26b4-3f5f-40d2-b866-ad3915e771bb nodeName:}" failed. No retries permitted until 2026-09-16 17:26:49.16976667 +0000 UTC m=+121.639764806 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/7a9b26b4-3f5f-40d2-b866-ad3915e771bb-service-ca-bundle") pod "router-default-99d9f4-g4nhp" (UID: "7a9b26b4-3f5f-40d2-b866-ad3915e771bb") : configmap references non-existent config key: service-ca.crt
Sep 16 17:26:48.670113 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:26:48.669824    2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 16 17:26:48.670113 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:26:48.669899    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7a9b26b4-3f5f-40d2-b866-ad3915e771bb-metrics-certs podName:7a9b26b4-3f5f-40d2-b866-ad3915e771bb nodeName:}" failed. No retries permitted until 2026-09-16 17:26:49.169883205 +0000 UTC m=+121.639881336 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7a9b26b4-3f5f-40d2-b866-ad3915e771bb-metrics-certs") pod "router-default-99d9f4-g4nhp" (UID: "7a9b26b4-3f5f-40d2-b866-ad3915e771bb") : secret "router-metrics-certs-default" not found
Sep 16 17:26:48.672107 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:26:48.672078    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/7a9b26b4-3f5f-40d2-b866-ad3915e771bb-stats-auth\") pod \"router-default-99d9f4-g4nhp\" (UID: \"7a9b26b4-3f5f-40d2-b866-ad3915e771bb\") " pod="openshift-ingress/router-default-99d9f4-g4nhp"
Sep 16 17:26:48.672107 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:26:48.672098    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/7a9b26b4-3f5f-40d2-b866-ad3915e771bb-default-certificate\") pod \"router-default-99d9f4-g4nhp\" (UID: \"7a9b26b4-3f5f-40d2-b866-ad3915e771bb\") " pod="openshift-ingress/router-default-99d9f4-g4nhp"
Sep 16 17:26:48.687768 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:26:48.687738    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6jsq\" (UniqueName: \"kubernetes.io/projected/7a9b26b4-3f5f-40d2-b866-ad3915e771bb-kube-api-access-k6jsq\") pod \"router-default-99d9f4-g4nhp\" (UID: \"7a9b26b4-3f5f-40d2-b866-ad3915e771bb\") " pod="openshift-ingress/router-default-99d9f4-g4nhp"
Sep 16 17:26:48.728979 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:26:48.728951    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-7v4db"]
Sep 16 17:26:48.730793 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:26:48.730777    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-7v4db"
Sep 16 17:26:48.733210 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:26:48.733187    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Sep 16 17:26:48.733402 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:26:48.733384    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-gwh46\""
Sep 16 17:26:48.733459 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:26:48.733421    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Sep 16 17:26:48.733521 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:26:48.733503    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Sep 16 17:26:48.733642 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:26:48.733629    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Sep 16 17:26:48.744465 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:26:48.744439    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-7v4db"]
Sep 16 17:26:48.771168 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:26:48.771139    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smfch\" (UniqueName: \"kubernetes.io/projected/51b86650-5910-4bc0-9832-dc967737cbea-kube-api-access-smfch\") pod \"kube-storage-version-migrator-operator-6c8f84f897-f7kcw\" (UID: \"51b86650-5910-4bc0-9832-dc967737cbea\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-f7kcw"
Sep 16 17:26:48.771330 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:26:48.771224    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-554hg\" (UniqueName: \"kubernetes.io/projected/4cf3f89a-3a7d-4651-94c3-26fe3ee314cc-kube-api-access-554hg\") pod \"service-ca-operator-cf7c776b7-7v4db\" (UID: \"4cf3f89a-3a7d-4651-94c3-26fe3ee314cc\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-7v4db"
Sep 16 17:26:48.771330 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:26:48.771245    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4cf3f89a-3a7d-4651-94c3-26fe3ee314cc-serving-cert\") pod \"service-ca-operator-cf7c776b7-7v4db\" (UID: \"4cf3f89a-3a7d-4651-94c3-26fe3ee314cc\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-7v4db"
Sep 16 17:26:48.771330 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:26:48.771261    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4cf3f89a-3a7d-4651-94c3-26fe3ee314cc-config\") pod \"service-ca-operator-cf7c776b7-7v4db\" (UID: \"4cf3f89a-3a7d-4651-94c3-26fe3ee314cc\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-7v4db"
Sep 16 17:26:48.771330 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:26:48.771293    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/51b86650-5910-4bc0-9832-dc967737cbea-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-f7kcw\" (UID: \"51b86650-5910-4bc0-9832-dc967737cbea\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-f7kcw"
Sep 16 17:26:48.771570 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:26:48.771362    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/51b86650-5910-4bc0-9832-dc967737cbea-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-f7kcw\" (UID: \"51b86650-5910-4bc0-9832-dc967737cbea\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-f7kcw"
Sep 16 17:26:48.771920 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:26:48.771898    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/51b86650-5910-4bc0-9832-dc967737cbea-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-f7kcw\" (UID: \"51b86650-5910-4bc0-9832-dc967737cbea\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-f7kcw"
Sep 16 17:26:48.773508 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:26:48.773489    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/51b86650-5910-4bc0-9832-dc967737cbea-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-f7kcw\" (UID: \"51b86650-5910-4bc0-9832-dc967737cbea\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-f7kcw"
Sep 16 17:26:48.785897 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:26:48.785861    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-smfch\" (UniqueName: \"kubernetes.io/projected/51b86650-5910-4bc0-9832-dc967737cbea-kube-api-access-smfch\") pod \"kube-storage-version-migrator-operator-6c8f84f897-f7kcw\" (UID: \"51b86650-5910-4bc0-9832-dc967737cbea\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-f7kcw"
Sep 16 17:26:48.831338 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:26:48.831304    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-8v98r"]
Sep 16 17:26:48.833303 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:26:48.833286    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-8v98r"
Sep 16 17:26:48.836115 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:26:48.836089    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Sep 16 17:26:48.836316 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:26:48.836166    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 16 17:26:48.838166 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:26:48.838143    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 16 17:26:48.838291 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:26:48.838273    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-hsppg\""
Sep 16 17:26:48.840539 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:26:48.839371    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Sep 16 17:26:48.845769 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:26:48.845743    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-kkgj6"]
Sep 16 17:26:48.847782 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:26:48.847759    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-8v98r"]
Sep 16 17:26:48.847894 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:26:48.847870    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f8b768455-kkgj6"
Sep 16 17:26:48.850773 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:26:48.850751    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-hfqxr\""
Sep 16 17:26:48.861172 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:26:48.861147    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-kkgj6"]
Sep 16 17:26:48.872680 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:26:48.872649    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a37645e3-e094-46da-992b-c0ea722bc83f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-8v98r\" (UID: \"a37645e3-e094-46da-992b-c0ea722bc83f\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-8v98r"
Sep 16 17:26:48.872806 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:26:48.872705    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4v48\" (UniqueName: \"kubernetes.io/projected/a37645e3-e094-46da-992b-c0ea722bc83f-kube-api-access-g4v48\") pod \"cluster-monitoring-operator-6cd954b897-8v98r\" (UID: \"a37645e3-e094-46da-992b-c0ea722bc83f\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-8v98r"
Sep 16 17:26:48.872806 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:26:48.872787    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-554hg\" (UniqueName: \"kubernetes.io/projected/4cf3f89a-3a7d-4651-94c3-26fe3ee314cc-kube-api-access-554hg\") pod \"service-ca-operator-cf7c776b7-7v4db\" (UID: \"4cf3f89a-3a7d-4651-94c3-26fe3ee314cc\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-7v4db"
Sep 16 17:26:48.872935 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:26:48.872816    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/a37645e3-e094-46da-992b-c0ea722bc83f-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-8v98r\" (UID: \"a37645e3-e094-46da-992b-c0ea722bc83f\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-8v98r"
Sep 16 17:26:48.872935 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:26:48.872861    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4cf3f89a-3a7d-4651-94c3-26fe3ee314cc-serving-cert\") pod \"service-ca-operator-cf7c776b7-7v4db\" (UID: \"4cf3f89a-3a7d-4651-94c3-26fe3ee314cc\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-7v4db"
Sep 16 17:26:48.872935 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:26:48.872888    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4cf3f89a-3a7d-4651-94c3-26fe3ee314cc-config\") pod \"service-ca-operator-cf7c776b7-7v4db\" (UID: \"4cf3f89a-3a7d-4651-94c3-26fe3ee314cc\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-7v4db"
Sep 16 17:26:48.873390 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:26:48.873370    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4cf3f89a-3a7d-4651-94c3-26fe3ee314cc-config\") pod \"service-ca-operator-cf7c776b7-7v4db\" (UID: \"4cf3f89a-3a7d-4651-94c3-26fe3ee314cc\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-7v4db"
Sep 16 17:26:48.875086 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:26:48.875070    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4cf3f89a-3a7d-4651-94c3-26fe3ee314cc-serving-cert\") pod \"service-ca-operator-cf7c776b7-7v4db\" (UID: \"4cf3f89a-3a7d-4651-94c3-26fe3ee314cc\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-7v4db"
Sep 16 17:26:48.891570 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:26:48.891545    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-554hg\" (UniqueName: \"kubernetes.io/projected/4cf3f89a-3a7d-4651-94c3-26fe3ee314cc-kube-api-access-554hg\") pod \"service-ca-operator-cf7c776b7-7v4db\" (UID: \"4cf3f89a-3a7d-4651-94c3-26fe3ee314cc\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-7v4db"
Sep 16 17:26:48.945790 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:26:48.945755    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-f7kcw"
Sep 16 17:26:48.973726 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:26:48.973694    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a37645e3-e094-46da-992b-c0ea722bc83f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-8v98r\" (UID: \"a37645e3-e094-46da-992b-c0ea722bc83f\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-8v98r"
Sep 16 17:26:48.973890 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:26:48.973747    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4v48\" (UniqueName: \"kubernetes.io/projected/a37645e3-e094-46da-992b-c0ea722bc83f-kube-api-access-g4v48\") pod \"cluster-monitoring-operator-6cd954b897-8v98r\" (UID: \"a37645e3-e094-46da-992b-c0ea722bc83f\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-8v98r"
Sep 16 17:26:48.973890 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:26:48.973804    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/a37645e3-e094-46da-992b-c0ea722bc83f-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-8v98r\" (UID: \"a37645e3-e094-46da-992b-c0ea722bc83f\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-8v98r"
Sep 16 17:26:48.973890 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:26:48.973862    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4d2kr\" (UniqueName: \"kubernetes.io/projected/9e2b3ecc-4322-4ffd-b200-c06d12de516a-kube-api-access-4d2kr\") pod \"network-check-source-6f8b768455-kkgj6\" (UID: \"9e2b3ecc-4322-4ffd-b200-c06d12de516a\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-kkgj6"
Sep 16 17:26:48.974048 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:26:48.973863    2579 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 16 17:26:48.974048 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:26:48.973960    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a37645e3-e094-46da-992b-c0ea722bc83f-cluster-monitoring-operator-tls podName:a37645e3-e094-46da-992b-c0ea722bc83f nodeName:}" failed. No retries permitted until 2026-09-16 17:26:49.473939557 +0000 UTC m=+121.943937675 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a37645e3-e094-46da-992b-c0ea722bc83f-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-8v98r" (UID: "a37645e3-e094-46da-992b-c0ea722bc83f") : secret "cluster-monitoring-operator-tls" not found
Sep 16 17:26:48.974510 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:26:48.974489    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/a37645e3-e094-46da-992b-c0ea722bc83f-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-8v98r\" (UID: \"a37645e3-e094-46da-992b-c0ea722bc83f\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-8v98r"
Sep 16 17:26:48.984245 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:26:48.984220    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4v48\" (UniqueName: \"kubernetes.io/projected/a37645e3-e094-46da-992b-c0ea722bc83f-kube-api-access-g4v48\") pod \"cluster-monitoring-operator-6cd954b897-8v98r\" (UID: \"a37645e3-e094-46da-992b-c0ea722bc83f\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-8v98r"
Sep 16 17:26:49.039473 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:26:49.039391    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-7v4db"
Sep 16 17:26:49.074639 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:26:49.074605    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4d2kr\" (UniqueName: \"kubernetes.io/projected/9e2b3ecc-4322-4ffd-b200-c06d12de516a-kube-api-access-4d2kr\") pod \"network-check-source-6f8b768455-kkgj6\" (UID: \"9e2b3ecc-4322-4ffd-b200-c06d12de516a\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-kkgj6"
Sep 16 17:26:49.084025 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:26:49.083967    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4d2kr\" (UniqueName: \"kubernetes.io/projected/9e2b3ecc-4322-4ffd-b200-c06d12de516a-kube-api-access-4d2kr\") pod \"network-check-source-6f8b768455-kkgj6\" (UID: \"9e2b3ecc-4322-4ffd-b200-c06d12de516a\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-kkgj6"
Sep 16 17:26:49.155815 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:26:49.155771    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f8b768455-kkgj6"
Sep 16 17:26:49.175305 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:26:49.175271    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7a9b26b4-3f5f-40d2-b866-ad3915e771bb-service-ca-bundle\") pod \"router-default-99d9f4-g4nhp\" (UID: \"7a9b26b4-3f5f-40d2-b866-ad3915e771bb\") " pod="openshift-ingress/router-default-99d9f4-g4nhp"
Sep 16 17:26:49.175479 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:26:49.175319    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a9b26b4-3f5f-40d2-b866-ad3915e771bb-metrics-certs\") pod \"router-default-99d9f4-g4nhp\" (UID: \"7a9b26b4-3f5f-40d2-b866-ad3915e771bb\") " pod="openshift-ingress/router-default-99d9f4-g4nhp"
Sep 16 17:26:49.175479 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:26:49.175436    2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 16 17:26:49.175479 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:26:49.175446    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7a9b26b4-3f5f-40d2-b866-ad3915e771bb-service-ca-bundle podName:7a9b26b4-3f5f-40d2-b866-ad3915e771bb nodeName:}" failed. No retries permitted until 2026-09-16 17:26:50.175427314 +0000 UTC m=+122.645425432 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/7a9b26b4-3f5f-40d2-b866-ad3915e771bb-service-ca-bundle") pod "router-default-99d9f4-g4nhp" (UID: "7a9b26b4-3f5f-40d2-b866-ad3915e771bb") : configmap references non-existent config key: service-ca.crt
Sep 16 17:26:49.175479 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:26:49.175476    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7a9b26b4-3f5f-40d2-b866-ad3915e771bb-metrics-certs podName:7a9b26b4-3f5f-40d2-b866-ad3915e771bb nodeName:}" failed. No retries permitted until 2026-09-16 17:26:50.175464359 +0000 UTC m=+122.645462472 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7a9b26b4-3f5f-40d2-b866-ad3915e771bb-metrics-certs") pod "router-default-99d9f4-g4nhp" (UID: "7a9b26b4-3f5f-40d2-b866-ad3915e771bb") : secret "router-metrics-certs-default" not found
Sep 16 17:26:49.215733 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:26:49.215706    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51b86650_5910_4bc0_9832_dc967737cbea.slice/crio-6b0a309c982845d5d086ef9ba3e922735ce3decc59b06add486b006da4199f20 WatchSource:0}: Error finding container 6b0a309c982845d5d086ef9ba3e922735ce3decc59b06add486b006da4199f20: Status 404 returned error can't find the container with id 6b0a309c982845d5d086ef9ba3e922735ce3decc59b06add486b006da4199f20
Sep 16 17:26:49.219669 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:26:49.219599    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-f7kcw"]
Sep 16 17:26:49.306477 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:26:49.306394    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-7v4db"]
Sep 16 17:26:49.309192 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:26:49.309159    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4cf3f89a_3a7d_4651_94c3_26fe3ee314cc.slice/crio-6d54393d6d31b213971a2f8d2270a22b3fb39566a2e3e540038fc3cc6ff130ed WatchSource:0}: Error finding container 6d54393d6d31b213971a2f8d2270a22b3fb39566a2e3e540038fc3cc6ff130ed: Status 404 returned error can't find the container with id 6d54393d6d31b213971a2f8d2270a22b3fb39566a2e3e540038fc3cc6ff130ed
Sep 16 17:26:49.422788 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:26:49.422757    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-kkgj6"]
Sep 16 17:26:49.425637 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:26:49.425608    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e2b3ecc_4322_4ffd_b200_c06d12de516a.slice/crio-15b2fe977fd530d37469339c19852df75f91092986e2c75f8f4819c1f8a33eb9 WatchSource:0}: Error finding container 15b2fe977fd530d37469339c19852df75f91092986e2c75f8f4819c1f8a33eb9: Status 404 returned error can't find the container with id 15b2fe977fd530d37469339c19852df75f91092986e2c75f8f4819c1f8a33eb9
Sep 16 17:26:49.477458 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:26:49.477426    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a37645e3-e094-46da-992b-c0ea722bc83f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-8v98r\" (UID: \"a37645e3-e094-46da-992b-c0ea722bc83f\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-8v98r"
Sep 16 17:26:49.477597 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:26:49.477579    2579 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 16 17:26:49.477657 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:26:49.477647    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a37645e3-e094-46da-992b-c0ea722bc83f-cluster-monitoring-operator-tls podName:a37645e3-e094-46da-992b-c0ea722bc83f nodeName:}" failed. No retries permitted until 2026-09-16 17:26:50.477625456 +0000 UTC m=+122.947623576 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a37645e3-e094-46da-992b-c0ea722bc83f-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-8v98r" (UID: "a37645e3-e094-46da-992b-c0ea722bc83f") : secret "cluster-monitoring-operator-tls" not found
Sep 16 17:26:49.496181 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:26:49.496150    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f8b768455-kkgj6" event={"ID":"9e2b3ecc-4322-4ffd-b200-c06d12de516a","Type":"ContainerStarted","Data":"77c405c806cbf748b30d49dd18fa5742eb6d2edf4f3db1b2182e923681d4ddfb"}
Sep 16 17:26:49.496350 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:26:49.496199    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f8b768455-kkgj6" event={"ID":"9e2b3ecc-4322-4ffd-b200-c06d12de516a","Type":"ContainerStarted","Data":"15b2fe977fd530d37469339c19852df75f91092986e2c75f8f4819c1f8a33eb9"}
Sep 16 17:26:49.497325 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:26:49.497302    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-f7kcw" event={"ID":"51b86650-5910-4bc0-9832-dc967737cbea","Type":"ContainerStarted","Data":"6b0a309c982845d5d086ef9ba3e922735ce3decc59b06add486b006da4199f20"}
Sep 16 17:26:49.498384 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:26:49.498363    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-7v4db" event={"ID":"4cf3f89a-3a7d-4651-94c3-26fe3ee314cc","Type":"ContainerStarted","Data":"6d54393d6d31b213971a2f8d2270a22b3fb39566a2e3e540038fc3cc6ff130ed"}
Sep 16 17:26:49.515482 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:26:49.515432    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f8b768455-kkgj6" podStartSLOduration=1.5154156410000001 podStartE2EDuration="1.515415641s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:26:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:26:49.514288403 +0000 UTC m=+121.984286534" watchObservedRunningTime="2026-09-16 17:26:49.515415641 +0000 UTC m=+121.985413777"
Sep 16 17:26:50.183010 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:26:50.182974    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7a9b26b4-3f5f-40d2-b866-ad3915e771bb-service-ca-bundle\") pod \"router-default-99d9f4-g4nhp\" (UID: \"7a9b26b4-3f5f-40d2-b866-ad3915e771bb\") " pod="openshift-ingress/router-default-99d9f4-g4nhp"
Sep 16 17:26:50.183472 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:26:50.183052    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a9b26b4-3f5f-40d2-b866-ad3915e771bb-metrics-certs\") pod \"router-default-99d9f4-g4nhp\" (UID: \"7a9b26b4-3f5f-40d2-b866-ad3915e771bb\") " pod="openshift-ingress/router-default-99d9f4-g4nhp"
Sep 16 17:26:50.183472 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:26:50.183209    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7a9b26b4-3f5f-40d2-b866-ad3915e771bb-service-ca-bundle podName:7a9b26b4-3f5f-40d2-b866-ad3915e771bb nodeName:}" failed. No retries permitted until 2026-09-16 17:26:52.183184129 +0000 UTC m=+124.653182259 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/7a9b26b4-3f5f-40d2-b866-ad3915e771bb-service-ca-bundle") pod "router-default-99d9f4-g4nhp" (UID: "7a9b26b4-3f5f-40d2-b866-ad3915e771bb") : configmap references non-existent config key: service-ca.crt
Sep 16 17:26:50.183472 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:26:50.183209    2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 16 17:26:50.183472 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:26:50.183270    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7a9b26b4-3f5f-40d2-b866-ad3915e771bb-metrics-certs podName:7a9b26b4-3f5f-40d2-b866-ad3915e771bb nodeName:}" failed. No retries permitted until 2026-09-16 17:26:52.183259702 +0000 UTC m=+124.653257817 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7a9b26b4-3f5f-40d2-b866-ad3915e771bb-metrics-certs") pod "router-default-99d9f4-g4nhp" (UID: "7a9b26b4-3f5f-40d2-b866-ad3915e771bb") : secret "router-metrics-certs-default" not found
Sep 16 17:26:50.485483 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:26:50.485384    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a37645e3-e094-46da-992b-c0ea722bc83f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-8v98r\" (UID: \"a37645e3-e094-46da-992b-c0ea722bc83f\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-8v98r"
Sep 16 17:26:50.485650 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:26:50.485563    2579 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 16 17:26:50.485650 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:26:50.485645    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a37645e3-e094-46da-992b-c0ea722bc83f-cluster-monitoring-operator-tls podName:a37645e3-e094-46da-992b-c0ea722bc83f nodeName:}" failed. No retries permitted until 2026-09-16 17:26:52.485625291 +0000 UTC m=+124.955623410 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a37645e3-e094-46da-992b-c0ea722bc83f-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-8v98r" (UID: "a37645e3-e094-46da-992b-c0ea722bc83f") : secret "cluster-monitoring-operator-tls" not found
Sep 16 17:26:52.200871 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:26:52.200739    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a9b26b4-3f5f-40d2-b866-ad3915e771bb-metrics-certs\") pod \"router-default-99d9f4-g4nhp\" (UID: \"7a9b26b4-3f5f-40d2-b866-ad3915e771bb\") " pod="openshift-ingress/router-default-99d9f4-g4nhp"
Sep 16 17:26:52.200871 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:26:52.200860    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7a9b26b4-3f5f-40d2-b866-ad3915e771bb-service-ca-bundle\") pod \"router-default-99d9f4-g4nhp\" (UID: \"7a9b26b4-3f5f-40d2-b866-ad3915e771bb\") " pod="openshift-ingress/router-default-99d9f4-g4nhp"
Sep 16 17:26:52.201292 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:26:52.200928    2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 16 17:26:52.201292 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:26:52.201013    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7a9b26b4-3f5f-40d2-b866-ad3915e771bb-metrics-certs podName:7a9b26b4-3f5f-40d2-b866-ad3915e771bb nodeName:}" failed. No retries permitted until 2026-09-16 17:26:56.200990386 +0000 UTC m=+128.670988501 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7a9b26b4-3f5f-40d2-b866-ad3915e771bb-metrics-certs") pod "router-default-99d9f4-g4nhp" (UID: "7a9b26b4-3f5f-40d2-b866-ad3915e771bb") : secret "router-metrics-certs-default" not found
Sep 16 17:26:52.201292 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:26:52.201040    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7a9b26b4-3f5f-40d2-b866-ad3915e771bb-service-ca-bundle podName:7a9b26b4-3f5f-40d2-b866-ad3915e771bb nodeName:}" failed. No retries permitted until 2026-09-16 17:26:56.201026879 +0000 UTC m=+128.671025012 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/7a9b26b4-3f5f-40d2-b866-ad3915e771bb-service-ca-bundle") pod "router-default-99d9f4-g4nhp" (UID: "7a9b26b4-3f5f-40d2-b866-ad3915e771bb") : configmap references non-existent config key: service-ca.crt
Sep 16 17:26:52.503432 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:26:52.503339    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a37645e3-e094-46da-992b-c0ea722bc83f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-8v98r\" (UID: \"a37645e3-e094-46da-992b-c0ea722bc83f\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-8v98r"
Sep 16 17:26:52.503599 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:26:52.503541    2579 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 16 17:26:52.503662 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:26:52.503628    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a37645e3-e094-46da-992b-c0ea722bc83f-cluster-monitoring-operator-tls podName:a37645e3-e094-46da-992b-c0ea722bc83f nodeName:}" failed. No retries permitted until 2026-09-16 17:26:56.503606496 +0000 UTC m=+128.973604611 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a37645e3-e094-46da-992b-c0ea722bc83f-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-8v98r" (UID: "a37645e3-e094-46da-992b-c0ea722bc83f") : secret "cluster-monitoring-operator-tls" not found
Sep 16 17:26:52.513980 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:26:52.513946    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-7v4db" event={"ID":"4cf3f89a-3a7d-4651-94c3-26fe3ee314cc","Type":"ContainerStarted","Data":"f10827809f462359605a2ce4fce62103c365c65855b44f6da745afbac68a1398"}
Sep 16 17:26:52.515557 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:26:52.515530    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-f7kcw" event={"ID":"51b86650-5910-4bc0-9832-dc967737cbea","Type":"ContainerStarted","Data":"8243e8741a4c7e81c281709658a5bab4b246c456435fa1cd3eebd430a8f28afc"}
Sep 16 17:26:52.531652 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:26:52.530586    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-7v4db" podStartSLOduration=1.9418999000000001 podStartE2EDuration="4.530566076s" totalImagesPullingTime="2.588666176s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:26:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:26:52.528195686 +0000 UTC m=+124.998193824" watchObservedRunningTime="2026-09-16 17:26:52.530566076 +0000 UTC m=+125.000564213"
Sep 16 17:26:52.546171 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:26:52.546123    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-f7kcw" podStartSLOduration=1.864017166 podStartE2EDuration="4.546106959s" totalImagesPullingTime="2.682089793s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:26:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:26:52.545354757 +0000 UTC m=+125.015352892" watchObservedRunningTime="2026-09-16 17:26:52.546106959 +0000 UTC m=+125.016105093"
Sep 16 17:26:53.190858 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:26:53.190809    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-zhxvb"]
Sep 16 17:26:53.193090 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:26:53.193075    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-zhxvb"
Sep 16 17:26:53.195444 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:26:53.195418    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 16 17:26:53.195573 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:26:53.195451    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 16 17:26:53.195573 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:26:53.195471    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 16 17:26:53.196448 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:26:53.196431    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-mfnhp\""
Sep 16 17:26:53.196561 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:26:53.196435    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 16 17:26:53.203477 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:26:53.203456    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-zhxvb"]
Sep 16 17:26:53.311156 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:26:53.311115    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3b4699f3-b8d0-408c-9031-37f1ec7cd316-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zhxvb\" (UID: \"3b4699f3-b8d0-408c-9031-37f1ec7cd316\") " pod="openshift-insights/insights-runtime-extractor-zhxvb"
Sep 16 17:26:53.311295 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:26:53.311174    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3b4699f3-b8d0-408c-9031-37f1ec7cd316-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zhxvb\" (UID: \"3b4699f3-b8d0-408c-9031-37f1ec7cd316\") " pod="openshift-insights/insights-runtime-extractor-zhxvb"
Sep 16 17:26:53.311295 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:26:53.311237    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3b4699f3-b8d0-408c-9031-37f1ec7cd316-crio-socket\") pod \"insights-runtime-extractor-zhxvb\" (UID: \"3b4699f3-b8d0-408c-9031-37f1ec7cd316\") " pod="openshift-insights/insights-runtime-extractor-zhxvb"
Sep 16 17:26:53.311295 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:26:53.311268    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rjzqj\" (UniqueName: \"kubernetes.io/projected/3b4699f3-b8d0-408c-9031-37f1ec7cd316-kube-api-access-rjzqj\") pod \"insights-runtime-extractor-zhxvb\" (UID: \"3b4699f3-b8d0-408c-9031-37f1ec7cd316\") " pod="openshift-insights/insights-runtime-extractor-zhxvb"
Sep 16 17:26:53.311409 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:26:53.311327    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3b4699f3-b8d0-408c-9031-37f1ec7cd316-data-volume\") pod \"insights-runtime-extractor-zhxvb\" (UID: \"3b4699f3-b8d0-408c-9031-37f1ec7cd316\") " pod="openshift-insights/insights-runtime-extractor-zhxvb"
Sep 16 17:26:53.411665 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:26:53.411631    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3b4699f3-b8d0-408c-9031-37f1ec7cd316-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zhxvb\" (UID: \"3b4699f3-b8d0-408c-9031-37f1ec7cd316\") " pod="openshift-insights/insights-runtime-extractor-zhxvb"
Sep 16 17:26:53.411861 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:26:53.411673    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3b4699f3-b8d0-408c-9031-37f1ec7cd316-crio-socket\") pod \"insights-runtime-extractor-zhxvb\" (UID: \"3b4699f3-b8d0-408c-9031-37f1ec7cd316\") " pod="openshift-insights/insights-runtime-extractor-zhxvb"
Sep 16 17:26:53.411861 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:26:53.411697    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjzqj\" (UniqueName: \"kubernetes.io/projected/3b4699f3-b8d0-408c-9031-37f1ec7cd316-kube-api-access-rjzqj\") pod \"insights-runtime-extractor-zhxvb\" (UID: \"3b4699f3-b8d0-408c-9031-37f1ec7cd316\") " pod="openshift-insights/insights-runtime-extractor-zhxvb"
Sep 16 17:26:53.411861 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:26:53.411774    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3b4699f3-b8d0-408c-9031-37f1ec7cd316-crio-socket\") pod \"insights-runtime-extractor-zhxvb\" (UID: \"3b4699f3-b8d0-408c-9031-37f1ec7cd316\") " pod="openshift-insights/insights-runtime-extractor-zhxvb"
Sep 16 17:26:53.411861 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:26:53.411794    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 17:26:53.412015 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:26:53.411871    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b4699f3-b8d0-408c-9031-37f1ec7cd316-insights-runtime-extractor-tls podName:3b4699f3-b8d0-408c-9031-37f1ec7cd316 nodeName:}" failed. No retries permitted until 2026-09-16 17:26:53.911839379 +0000 UTC m=+126.381837509 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3b4699f3-b8d0-408c-9031-37f1ec7cd316-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zhxvb" (UID: "3b4699f3-b8d0-408c-9031-37f1ec7cd316") : secret "insights-runtime-extractor-tls" not found
Sep 16 17:26:53.412015 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:26:53.411895    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3b4699f3-b8d0-408c-9031-37f1ec7cd316-data-volume\") pod \"insights-runtime-extractor-zhxvb\" (UID: \"3b4699f3-b8d0-408c-9031-37f1ec7cd316\") " pod="openshift-insights/insights-runtime-extractor-zhxvb"
Sep 16 17:26:53.412015 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:26:53.411967    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3b4699f3-b8d0-408c-9031-37f1ec7cd316-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zhxvb\" (UID: \"3b4699f3-b8d0-408c-9031-37f1ec7cd316\") " pod="openshift-insights/insights-runtime-extractor-zhxvb"
Sep 16 17:26:53.412123 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:26:53.412075    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 17:26:53.412170 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:26:53.412125    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3b4699f3-b8d0-408c-9031-37f1ec7cd316-kube-rbac-proxy-cm podName:3b4699f3-b8d0-408c-9031-37f1ec7cd316 nodeName:}" failed. No retries permitted until 2026-09-16 17:26:53.912110337 +0000 UTC m=+126.382108467 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/3b4699f3-b8d0-408c-9031-37f1ec7cd316-kube-rbac-proxy-cm") pod "insights-runtime-extractor-zhxvb" (UID: "3b4699f3-b8d0-408c-9031-37f1ec7cd316") : configmap "kube-rbac-proxy" not found
Sep 16 17:26:53.412278 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:26:53.412249    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3b4699f3-b8d0-408c-9031-37f1ec7cd316-data-volume\") pod \"insights-runtime-extractor-zhxvb\" (UID: \"3b4699f3-b8d0-408c-9031-37f1ec7cd316\") " pod="openshift-insights/insights-runtime-extractor-zhxvb"
Sep 16 17:26:53.413473 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:26:53.413442    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-vdcxh"]
Sep 16 17:26:53.415540 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:26:53.415524    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-vdcxh"
Sep 16 17:26:53.418048 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:26:53.418030    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-vlhd9\""
Sep 16 17:26:53.418162 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:26:53.418031    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Sep 16 17:26:53.418162 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:26:53.418062    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Sep 16 17:26:53.425471 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:26:53.425405    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-vdcxh"]
Sep 16 17:26:53.427818 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:26:53.427787    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjzqj\" (UniqueName: \"kubernetes.io/projected/3b4699f3-b8d0-408c-9031-37f1ec7cd316-kube-api-access-rjzqj\") pod \"insights-runtime-extractor-zhxvb\" (UID: \"3b4699f3-b8d0-408c-9031-37f1ec7cd316\") " pod="openshift-insights/insights-runtime-extractor-zhxvb"
Sep 16 17:26:53.513280 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:26:53.513184    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vq8pg\" (UniqueName: \"kubernetes.io/projected/4a2652f6-34d3-4fe8-82ec-2501aa923133-kube-api-access-vq8pg\") pod \"migrator-8b4d697c5-vdcxh\" (UID: \"4a2652f6-34d3-4fe8-82ec-2501aa923133\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-vdcxh"
Sep 16 17:26:53.613801 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:26:53.613761    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vq8pg\" (UniqueName: \"kubernetes.io/projected/4a2652f6-34d3-4fe8-82ec-2501aa923133-kube-api-access-vq8pg\") pod \"migrator-8b4d697c5-vdcxh\" (UID: \"4a2652f6-34d3-4fe8-82ec-2501aa923133\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-vdcxh"
Sep 16 17:26:53.622108 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:26:53.622076    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vq8pg\" (UniqueName: \"kubernetes.io/projected/4a2652f6-34d3-4fe8-82ec-2501aa923133-kube-api-access-vq8pg\") pod \"migrator-8b4d697c5-vdcxh\" (UID: \"4a2652f6-34d3-4fe8-82ec-2501aa923133\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-vdcxh"
Sep 16 17:26:53.724585 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:26:53.724544    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-vdcxh"
Sep 16 17:26:53.916696 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:26:53.916656    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3b4699f3-b8d0-408c-9031-37f1ec7cd316-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zhxvb\" (UID: \"3b4699f3-b8d0-408c-9031-37f1ec7cd316\") " pod="openshift-insights/insights-runtime-extractor-zhxvb"
Sep 16 17:26:53.916995 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:26:53.916721    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3b4699f3-b8d0-408c-9031-37f1ec7cd316-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zhxvb\" (UID: \"3b4699f3-b8d0-408c-9031-37f1ec7cd316\") " pod="openshift-insights/insights-runtime-extractor-zhxvb"
Sep 16 17:26:53.916995 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:26:53.916774    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 17:26:53.916995 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:26:53.916840    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3b4699f3-b8d0-408c-9031-37f1ec7cd316-kube-rbac-proxy-cm podName:3b4699f3-b8d0-408c-9031-37f1ec7cd316 nodeName:}" failed. No retries permitted until 2026-09-16 17:26:54.916822649 +0000 UTC m=+127.386820779 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/3b4699f3-b8d0-408c-9031-37f1ec7cd316-kube-rbac-proxy-cm") pod "insights-runtime-extractor-zhxvb" (UID: "3b4699f3-b8d0-408c-9031-37f1ec7cd316") : configmap "kube-rbac-proxy" not found
Sep 16 17:26:53.916995 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:26:53.916881    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 17:26:53.916995 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:26:53.916941    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b4699f3-b8d0-408c-9031-37f1ec7cd316-insights-runtime-extractor-tls podName:3b4699f3-b8d0-408c-9031-37f1ec7cd316 nodeName:}" failed. No retries permitted until 2026-09-16 17:26:54.916924679 +0000 UTC m=+127.386922809 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3b4699f3-b8d0-408c-9031-37f1ec7cd316-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zhxvb" (UID: "3b4699f3-b8d0-408c-9031-37f1ec7cd316") : secret "insights-runtime-extractor-tls" not found
Sep 16 17:26:53.989081 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:26:53.988943    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-vdcxh"]
Sep 16 17:26:53.991445 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:26:53.991420    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a2652f6_34d3_4fe8_82ec_2501aa923133.slice/crio-d56c69ede0f1e22ca4ef1ecd3d69359e8c6412d9dff9fffb9274ec23888e17d4 WatchSource:0}: Error finding container d56c69ede0f1e22ca4ef1ecd3d69359e8c6412d9dff9fffb9274ec23888e17d4: Status 404 returned error can't find the container with id d56c69ede0f1e22ca4ef1ecd3d69359e8c6412d9dff9fffb9274ec23888e17d4
Sep 16 17:26:54.522889 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:26:54.522839    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-vdcxh" event={"ID":"4a2652f6-34d3-4fe8-82ec-2501aa923133","Type":"ContainerStarted","Data":"d56c69ede0f1e22ca4ef1ecd3d69359e8c6412d9dff9fffb9274ec23888e17d4"}
Sep 16 17:26:54.925319 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:26:54.925275    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3b4699f3-b8d0-408c-9031-37f1ec7cd316-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zhxvb\" (UID: \"3b4699f3-b8d0-408c-9031-37f1ec7cd316\") " pod="openshift-insights/insights-runtime-extractor-zhxvb"
Sep 16 17:26:54.925523 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:26:54.925357    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3b4699f3-b8d0-408c-9031-37f1ec7cd316-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zhxvb\" (UID: \"3b4699f3-b8d0-408c-9031-37f1ec7cd316\") " pod="openshift-insights/insights-runtime-extractor-zhxvb"
Sep 16 17:26:54.925523 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:26:54.925387    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 17:26:54.925523 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:26:54.925481    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3b4699f3-b8d0-408c-9031-37f1ec7cd316-kube-rbac-proxy-cm podName:3b4699f3-b8d0-408c-9031-37f1ec7cd316 nodeName:}" failed. No retries permitted until 2026-09-16 17:26:56.925459218 +0000 UTC m=+129.395457335 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/3b4699f3-b8d0-408c-9031-37f1ec7cd316-kube-rbac-proxy-cm") pod "insights-runtime-extractor-zhxvb" (UID: "3b4699f3-b8d0-408c-9031-37f1ec7cd316") : configmap "kube-rbac-proxy" not found
Sep 16 17:26:54.925523 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:26:54.925488    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 17:26:54.925690 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:26:54.925540    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b4699f3-b8d0-408c-9031-37f1ec7cd316-insights-runtime-extractor-tls podName:3b4699f3-b8d0-408c-9031-37f1ec7cd316 nodeName:}" failed. No retries permitted until 2026-09-16 17:26:56.925524275 +0000 UTC m=+129.395522407 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3b4699f3-b8d0-408c-9031-37f1ec7cd316-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zhxvb" (UID: "3b4699f3-b8d0-408c-9031-37f1ec7cd316") : secret "insights-runtime-extractor-tls" not found
Sep 16 17:26:55.314633 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:26:55.314598    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-qtdvz_21a2db47-94e4-4d2b-9cb2-0b2bca64ef5b/dns-node-resolver/0.log"
Sep 16 17:26:55.527665 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:26:55.527568    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-vdcxh" event={"ID":"4a2652f6-34d3-4fe8-82ec-2501aa923133","Type":"ContainerStarted","Data":"627b5ccef29c06cb2e288ece44666ca589197cfcab5c3133a2187ca619a278a0"}
Sep 16 17:26:55.527665 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:26:55.527651    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-vdcxh" event={"ID":"4a2652f6-34d3-4fe8-82ec-2501aa923133","Type":"ContainerStarted","Data":"b1a94d2e8fcac006b00d8e7bec2049c1cd8a23917c5e6e245f38121f970efa0e"}
Sep 16 17:26:55.545810 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:26:55.545751    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-vdcxh" podStartSLOduration=1.370979961 podStartE2EDuration="2.545738057s" totalImagesPullingTime="1.174758096s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:26:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:26:55.544995017 +0000 UTC m=+128.014993153" watchObservedRunningTime="2026-09-16 17:26:55.545738057 +0000 UTC m=+128.015736241"
Sep 16 17:26:56.236856 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:26:56.236801    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a9b26b4-3f5f-40d2-b866-ad3915e771bb-metrics-certs\") pod \"router-default-99d9f4-g4nhp\" (UID: \"7a9b26b4-3f5f-40d2-b866-ad3915e771bb\") " pod="openshift-ingress/router-default-99d9f4-g4nhp"
Sep 16 17:26:56.237047 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:26:56.236936    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7a9b26b4-3f5f-40d2-b866-ad3915e771bb-service-ca-bundle\") pod \"router-default-99d9f4-g4nhp\" (UID: \"7a9b26b4-3f5f-40d2-b866-ad3915e771bb\") " pod="openshift-ingress/router-default-99d9f4-g4nhp"
Sep 16 17:26:56.237047 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:26:56.236965    2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 16 17:26:56.237047 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:26:56.237031    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7a9b26b4-3f5f-40d2-b866-ad3915e771bb-metrics-certs podName:7a9b26b4-3f5f-40d2-b866-ad3915e771bb nodeName:}" failed. No retries permitted until 2026-09-16 17:27:04.237012657 +0000 UTC m=+136.707010793 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7a9b26b4-3f5f-40d2-b866-ad3915e771bb-metrics-certs") pod "router-default-99d9f4-g4nhp" (UID: "7a9b26b4-3f5f-40d2-b866-ad3915e771bb") : secret "router-metrics-certs-default" not found
Sep 16 17:26:56.237047 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:26:56.237047    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7a9b26b4-3f5f-40d2-b866-ad3915e771bb-service-ca-bundle podName:7a9b26b4-3f5f-40d2-b866-ad3915e771bb nodeName:}" failed. No retries permitted until 2026-09-16 17:27:04.237040443 +0000 UTC m=+136.707038557 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/7a9b26b4-3f5f-40d2-b866-ad3915e771bb-service-ca-bundle") pod "router-default-99d9f4-g4nhp" (UID: "7a9b26b4-3f5f-40d2-b866-ad3915e771bb") : configmap references non-existent config key: service-ca.crt
Sep 16 17:26:56.315055 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:26:56.315027    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-t6cx6_41b48733-fd0e-4eaa-a91c-5d57f642b2f8/node-ca/0.log"
Sep 16 17:26:56.538693 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:26:56.538657    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a37645e3-e094-46da-992b-c0ea722bc83f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-8v98r\" (UID: \"a37645e3-e094-46da-992b-c0ea722bc83f\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-8v98r"
Sep 16 17:26:56.539123 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:26:56.538800    2579 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 16 17:26:56.539123 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:26:56.538884    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a37645e3-e094-46da-992b-c0ea722bc83f-cluster-monitoring-operator-tls podName:a37645e3-e094-46da-992b-c0ea722bc83f nodeName:}" failed. No retries permitted until 2026-09-16 17:27:04.538867115 +0000 UTC m=+137.008865231 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a37645e3-e094-46da-992b-c0ea722bc83f-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-8v98r" (UID: "a37645e3-e094-46da-992b-c0ea722bc83f") : secret "cluster-monitoring-operator-tls" not found
Sep 16 17:26:56.943130 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:26:56.943021    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3b4699f3-b8d0-408c-9031-37f1ec7cd316-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zhxvb\" (UID: \"3b4699f3-b8d0-408c-9031-37f1ec7cd316\") " pod="openshift-insights/insights-runtime-extractor-zhxvb"
Sep 16 17:26:56.943130 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:26:56.943096    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3b4699f3-b8d0-408c-9031-37f1ec7cd316-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zhxvb\" (UID: \"3b4699f3-b8d0-408c-9031-37f1ec7cd316\") " pod="openshift-insights/insights-runtime-extractor-zhxvb"
Sep 16 17:26:56.943361 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:26:56.943156    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 17:26:56.943361 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:26:56.943216    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 17:26:56.943361 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:26:56.943225    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3b4699f3-b8d0-408c-9031-37f1ec7cd316-kube-rbac-proxy-cm podName:3b4699f3-b8d0-408c-9031-37f1ec7cd316 nodeName:}" failed. No retries permitted until 2026-09-16 17:27:00.943209773 +0000 UTC m=+133.413207888 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/3b4699f3-b8d0-408c-9031-37f1ec7cd316-kube-rbac-proxy-cm") pod "insights-runtime-extractor-zhxvb" (UID: "3b4699f3-b8d0-408c-9031-37f1ec7cd316") : configmap "kube-rbac-proxy" not found
Sep 16 17:26:56.943361 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:26:56.943289    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b4699f3-b8d0-408c-9031-37f1ec7cd316-insights-runtime-extractor-tls podName:3b4699f3-b8d0-408c-9031-37f1ec7cd316 nodeName:}" failed. No retries permitted until 2026-09-16 17:27:00.943273416 +0000 UTC m=+133.413271533 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3b4699f3-b8d0-408c-9031-37f1ec7cd316-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zhxvb" (UID: "3b4699f3-b8d0-408c-9031-37f1ec7cd316") : secret "insights-runtime-extractor-tls" not found
Sep 16 17:26:57.115104 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:26:57.115053    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-t8qph_7886f4b3-cb86-4a0e-ae6c-809ac7a567fa/serve-healthcheck-canary/0.log"
Sep 16 17:26:57.851283 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:26:57.851227    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bc719829-66b1-4a84-b994-e71cf111dafa-metrics-certs\") pod \"network-metrics-daemon-xfgjq\" (UID: \"bc719829-66b1-4a84-b994-e71cf111dafa\") " pod="openshift-multus/network-metrics-daemon-xfgjq"
Sep 16 17:26:57.851681 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:26:57.851380    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 16 17:26:57.851681 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:26:57.851446    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc719829-66b1-4a84-b994-e71cf111dafa-metrics-certs podName:bc719829-66b1-4a84-b994-e71cf111dafa nodeName:}" failed. No retries permitted until 2026-09-16 17:28:59.851432142 +0000 UTC m=+252.321430254 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bc719829-66b1-4a84-b994-e71cf111dafa-metrics-certs") pod "network-metrics-daemon-xfgjq" (UID: "bc719829-66b1-4a84-b994-e71cf111dafa") : secret "metrics-daemon-secret" not found
Sep 16 17:27:00.979588 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:00.979529    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3b4699f3-b8d0-408c-9031-37f1ec7cd316-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zhxvb\" (UID: \"3b4699f3-b8d0-408c-9031-37f1ec7cd316\") " pod="openshift-insights/insights-runtime-extractor-zhxvb"
Sep 16 17:27:00.980022 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:00.979605    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3b4699f3-b8d0-408c-9031-37f1ec7cd316-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zhxvb\" (UID: \"3b4699f3-b8d0-408c-9031-37f1ec7cd316\") " pod="openshift-insights/insights-runtime-extractor-zhxvb"
Sep 16 17:27:00.980022 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:27:00.979656    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 17:27:00.980022 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:27:00.979719    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3b4699f3-b8d0-408c-9031-37f1ec7cd316-kube-rbac-proxy-cm podName:3b4699f3-b8d0-408c-9031-37f1ec7cd316 nodeName:}" failed. No retries permitted until 2026-09-16 17:27:08.979704627 +0000 UTC m=+141.449702740 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/3b4699f3-b8d0-408c-9031-37f1ec7cd316-kube-rbac-proxy-cm") pod "insights-runtime-extractor-zhxvb" (UID: "3b4699f3-b8d0-408c-9031-37f1ec7cd316") : configmap "kube-rbac-proxy" not found
Sep 16 17:27:00.980022 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:27:00.979735    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 17:27:00.980022 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:27:00.979780    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b4699f3-b8d0-408c-9031-37f1ec7cd316-insights-runtime-extractor-tls podName:3b4699f3-b8d0-408c-9031-37f1ec7cd316 nodeName:}" failed. No retries permitted until 2026-09-16 17:27:08.97976479 +0000 UTC m=+141.449762909 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3b4699f3-b8d0-408c-9031-37f1ec7cd316-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zhxvb" (UID: "3b4699f3-b8d0-408c-9031-37f1ec7cd316") : secret "insights-runtime-extractor-tls" not found
Sep 16 17:27:04.309152 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:04.309113    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7a9b26b4-3f5f-40d2-b866-ad3915e771bb-service-ca-bundle\") pod \"router-default-99d9f4-g4nhp\" (UID: \"7a9b26b4-3f5f-40d2-b866-ad3915e771bb\") " pod="openshift-ingress/router-default-99d9f4-g4nhp"
Sep 16 17:27:04.309548 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:04.309182    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a9b26b4-3f5f-40d2-b866-ad3915e771bb-metrics-certs\") pod \"router-default-99d9f4-g4nhp\" (UID: \"7a9b26b4-3f5f-40d2-b866-ad3915e771bb\") " pod="openshift-ingress/router-default-99d9f4-g4nhp"
Sep 16 17:27:04.309548 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:27:04.309291    2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 16 17:27:04.309548 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:27:04.309294    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7a9b26b4-3f5f-40d2-b866-ad3915e771bb-service-ca-bundle podName:7a9b26b4-3f5f-40d2-b866-ad3915e771bb nodeName:}" failed. No retries permitted until 2026-09-16 17:27:20.309276822 +0000 UTC m=+152.779274934 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/7a9b26b4-3f5f-40d2-b866-ad3915e771bb-service-ca-bundle") pod "router-default-99d9f4-g4nhp" (UID: "7a9b26b4-3f5f-40d2-b866-ad3915e771bb") : configmap references non-existent config key: service-ca.crt
Sep 16 17:27:04.309548 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:27:04.309344    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7a9b26b4-3f5f-40d2-b866-ad3915e771bb-metrics-certs podName:7a9b26b4-3f5f-40d2-b866-ad3915e771bb nodeName:}" failed. No retries permitted until 2026-09-16 17:27:20.309333195 +0000 UTC m=+152.779331307 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7a9b26b4-3f5f-40d2-b866-ad3915e771bb-metrics-certs") pod "router-default-99d9f4-g4nhp" (UID: "7a9b26b4-3f5f-40d2-b866-ad3915e771bb") : secret "router-metrics-certs-default" not found
Sep 16 17:27:04.612326 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:04.612219    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a37645e3-e094-46da-992b-c0ea722bc83f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-8v98r\" (UID: \"a37645e3-e094-46da-992b-c0ea722bc83f\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-8v98r"
Sep 16 17:27:04.612479 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:27:04.612373    2579 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 16 17:27:04.612479 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:27:04.612438    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a37645e3-e094-46da-992b-c0ea722bc83f-cluster-monitoring-operator-tls podName:a37645e3-e094-46da-992b-c0ea722bc83f nodeName:}" failed. No retries permitted until 2026-09-16 17:27:20.612422812 +0000 UTC m=+153.082420925 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a37645e3-e094-46da-992b-c0ea722bc83f-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-8v98r" (UID: "a37645e3-e094-46da-992b-c0ea722bc83f") : secret "cluster-monitoring-operator-tls" not found
Sep 16 17:27:09.053187 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:09.053125    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3b4699f3-b8d0-408c-9031-37f1ec7cd316-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zhxvb\" (UID: \"3b4699f3-b8d0-408c-9031-37f1ec7cd316\") " pod="openshift-insights/insights-runtime-extractor-zhxvb"
Sep 16 17:27:09.053693 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:09.053211    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3b4699f3-b8d0-408c-9031-37f1ec7cd316-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zhxvb\" (UID: \"3b4699f3-b8d0-408c-9031-37f1ec7cd316\") " pod="openshift-insights/insights-runtime-extractor-zhxvb"
Sep 16 17:27:09.053693 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:27:09.053335    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 17:27:09.053693 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:27:09.053408    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b4699f3-b8d0-408c-9031-37f1ec7cd316-insights-runtime-extractor-tls podName:3b4699f3-b8d0-408c-9031-37f1ec7cd316 nodeName:}" failed. No retries permitted until 2026-09-16 17:27:25.053387999 +0000 UTC m=+157.523386129 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3b4699f3-b8d0-408c-9031-37f1ec7cd316-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zhxvb" (UID: "3b4699f3-b8d0-408c-9031-37f1ec7cd316") : secret "insights-runtime-extractor-tls" not found
Sep 16 17:27:09.053693 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:09.053648    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3b4699f3-b8d0-408c-9031-37f1ec7cd316-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zhxvb\" (UID: \"3b4699f3-b8d0-408c-9031-37f1ec7cd316\") " pod="openshift-insights/insights-runtime-extractor-zhxvb"
Sep 16 17:27:17.690514 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:17.690473    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-76f99584f4-xf777"]
Sep 16 17:27:17.693421 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:17.693401    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-76f99584f4-xf777"
Sep 16 17:27:17.696036 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:17.696008    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 16 17:27:17.696162 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:17.696049    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 16 17:27:17.696768 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:17.696748    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-4bsx9\""
Sep 16 17:27:17.709997 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:17.709972    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-76f99584f4-xf777"]
Sep 16 17:27:17.818551 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:17.818518    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljklq\" (UniqueName: \"kubernetes.io/projected/ff8acb9f-034a-4aab-aed4-aebb3df42f07-kube-api-access-ljklq\") pod \"downloads-76f99584f4-xf777\" (UID: \"ff8acb9f-034a-4aab-aed4-aebb3df42f07\") " pod="openshift-console/downloads-76f99584f4-xf777"
Sep 16 17:27:17.919622 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:17.919586    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljklq\" (UniqueName: \"kubernetes.io/projected/ff8acb9f-034a-4aab-aed4-aebb3df42f07-kube-api-access-ljklq\") pod \"downloads-76f99584f4-xf777\" (UID: \"ff8acb9f-034a-4aab-aed4-aebb3df42f07\") " pod="openshift-console/downloads-76f99584f4-xf777"
Sep 16 17:27:17.932017 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:17.931985    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljklq\" (UniqueName: \"kubernetes.io/projected/ff8acb9f-034a-4aab-aed4-aebb3df42f07-kube-api-access-ljklq\") pod \"downloads-76f99584f4-xf777\" (UID: \"ff8acb9f-034a-4aab-aed4-aebb3df42f07\") " pod="openshift-console/downloads-76f99584f4-xf777"
Sep 16 17:27:18.002061 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:18.001968    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-76f99584f4-xf777"
Sep 16 17:27:18.281370 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:18.281342    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-76f99584f4-xf777"]
Sep 16 17:27:18.284559 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:27:18.284529    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff8acb9f_034a_4aab_aed4_aebb3df42f07.slice/crio-2144bac93f6292fa34fe6badcc621dddf0515805bb7adbac1e67551e51208650 WatchSource:0}: Error finding container 2144bac93f6292fa34fe6badcc621dddf0515805bb7adbac1e67551e51208650: Status 404 returned error can't find the container with id 2144bac93f6292fa34fe6badcc621dddf0515805bb7adbac1e67551e51208650
Sep 16 17:27:18.601504 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:18.601415    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-76f99584f4-xf777" event={"ID":"ff8acb9f-034a-4aab-aed4-aebb3df42f07","Type":"ContainerStarted","Data":"2144bac93f6292fa34fe6badcc621dddf0515805bb7adbac1e67551e51208650"}
Sep 16 17:27:20.338564 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:20.338524    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a9b26b4-3f5f-40d2-b866-ad3915e771bb-metrics-certs\") pod \"router-default-99d9f4-g4nhp\" (UID: \"7a9b26b4-3f5f-40d2-b866-ad3915e771bb\") " pod="openshift-ingress/router-default-99d9f4-g4nhp"
Sep 16 17:27:20.339031 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:20.338625    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7a9b26b4-3f5f-40d2-b866-ad3915e771bb-service-ca-bundle\") pod \"router-default-99d9f4-g4nhp\" (UID: \"7a9b26b4-3f5f-40d2-b866-ad3915e771bb\") " pod="openshift-ingress/router-default-99d9f4-g4nhp"
Sep 16 17:27:20.339296 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:20.339273    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7a9b26b4-3f5f-40d2-b866-ad3915e771bb-service-ca-bundle\") pod \"router-default-99d9f4-g4nhp\" (UID: \"7a9b26b4-3f5f-40d2-b866-ad3915e771bb\") " pod="openshift-ingress/router-default-99d9f4-g4nhp"
Sep 16 17:27:20.341212 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:20.341191    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a9b26b4-3f5f-40d2-b866-ad3915e771bb-metrics-certs\") pod \"router-default-99d9f4-g4nhp\" (UID: \"7a9b26b4-3f5f-40d2-b866-ad3915e771bb\") " pod="openshift-ingress/router-default-99d9f4-g4nhp"
Sep 16 17:27:20.638542 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:20.638458    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-99d9f4-g4nhp"
Sep 16 17:27:20.641326 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:20.641302    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a37645e3-e094-46da-992b-c0ea722bc83f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-8v98r\" (UID: \"a37645e3-e094-46da-992b-c0ea722bc83f\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-8v98r"
Sep 16 17:27:20.643570 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:20.643548    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a37645e3-e094-46da-992b-c0ea722bc83f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-8v98r\" (UID: \"a37645e3-e094-46da-992b-c0ea722bc83f\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-8v98r"
Sep 16 17:27:20.644405 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:20.644376    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-8v98r"
Sep 16 17:27:20.913337 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:20.913203    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-8v98r"]
Sep 16 17:27:20.916245 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:20.916217    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-99d9f4-g4nhp"]
Sep 16 17:27:20.916347 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:27:20.916274    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda37645e3_e094_46da_992b_c0ea722bc83f.slice/crio-c248efa619cee21adf52a6a8fd7868e127c3991306d2fec46ba01dae81fcd21a WatchSource:0}: Error finding container c248efa619cee21adf52a6a8fd7868e127c3991306d2fec46ba01dae81fcd21a: Status 404 returned error can't find the container with id c248efa619cee21adf52a6a8fd7868e127c3991306d2fec46ba01dae81fcd21a
Sep 16 17:27:20.919486 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:27:20.919461    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a9b26b4_3f5f_40d2_b866_ad3915e771bb.slice/crio-3c0581abb01a98d81bc2209104b03b081964132312eaf7f161eef92f0761281f WatchSource:0}: Error finding container 3c0581abb01a98d81bc2209104b03b081964132312eaf7f161eef92f0761281f: Status 404 returned error can't find the container with id 3c0581abb01a98d81bc2209104b03b081964132312eaf7f161eef92f0761281f
Sep 16 17:27:21.615288 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:21.615241    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-8v98r" event={"ID":"a37645e3-e094-46da-992b-c0ea722bc83f","Type":"ContainerStarted","Data":"c248efa619cee21adf52a6a8fd7868e127c3991306d2fec46ba01dae81fcd21a"}
Sep 16 17:27:21.617361 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:21.617175    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-99d9f4-g4nhp" event={"ID":"7a9b26b4-3f5f-40d2-b866-ad3915e771bb","Type":"ContainerStarted","Data":"b58ccee9f0f426ceb890e97cb045d19ff3301d8e4f57aded90192b34b4d5f0c8"}
Sep 16 17:27:21.617361 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:21.617231    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-99d9f4-g4nhp" event={"ID":"7a9b26b4-3f5f-40d2-b866-ad3915e771bb","Type":"ContainerStarted","Data":"3c0581abb01a98d81bc2209104b03b081964132312eaf7f161eef92f0761281f"}
Sep 16 17:27:21.638884 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:21.638833    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-99d9f4-g4nhp"
Sep 16 17:27:21.640486 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:21.640431    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-99d9f4-g4nhp" podStartSLOduration=33.640403698 podStartE2EDuration="33.640403698s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:26:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:27:21.639880926 +0000 UTC m=+154.109879062" watchObservedRunningTime="2026-09-16 17:27:21.640403698 +0000 UTC m=+154.110401837"
Sep 16 17:27:21.642046 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:21.641827    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-99d9f4-g4nhp"
Sep 16 17:27:22.623825 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:22.623745    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-8v98r" event={"ID":"a37645e3-e094-46da-992b-c0ea722bc83f","Type":"ContainerStarted","Data":"1d6b3611b4232c2ea50c671f781d2f67b223f0d01f40cab521addb4476cab2fd"}
Sep 16 17:27:22.624371 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:22.624319    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-99d9f4-g4nhp"
Sep 16 17:27:22.625633 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:22.625610    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-99d9f4-g4nhp"
Sep 16 17:27:22.679992 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:22.679932    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-8v98r" podStartSLOduration=33.127866711 podStartE2EDuration="34.679909149s" totalImagesPullingTime="1.552042438s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:26:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:27:22.678762102 +0000 UTC m=+155.148760239" watchObservedRunningTime="2026-09-16 17:27:22.679909149 +0000 UTC m=+155.149907299"
Sep 16 17:27:23.054670 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:23.054636    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-hw4m5"]
Sep 16 17:27:23.057953 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:23.057900    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-hw4m5"
Sep 16 17:27:23.061613 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:23.061586    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Sep 16 17:27:23.062322 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:23.062298    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-vkssh\""
Sep 16 17:27:23.072939 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:23.072906    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-hw4m5"]
Sep 16 17:27:23.162323 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:23.162282    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/21c42a22-b64c-4528-bd63-d5d583ba59ab-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-hw4m5\" (UID: \"21c42a22-b64c-4528-bd63-d5d583ba59ab\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-hw4m5"
Sep 16 17:27:23.263268 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:23.263229    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/21c42a22-b64c-4528-bd63-d5d583ba59ab-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-hw4m5\" (UID: \"21c42a22-b64c-4528-bd63-d5d583ba59ab\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-hw4m5"
Sep 16 17:27:23.263439 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:27:23.263409    2579 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found
Sep 16 17:27:23.263492 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:27:23.263482    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21c42a22-b64c-4528-bd63-d5d583ba59ab-tls-certificates podName:21c42a22-b64c-4528-bd63-d5d583ba59ab nodeName:}" failed. No retries permitted until 2026-09-16 17:27:23.763466515 +0000 UTC m=+156.233464632 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/21c42a22-b64c-4528-bd63-d5d583ba59ab-tls-certificates") pod "prometheus-operator-admission-webhook-5864cd99d7-hw4m5" (UID: "21c42a22-b64c-4528-bd63-d5d583ba59ab") : secret "prometheus-operator-admission-webhook-tls" not found
Sep 16 17:27:23.768477 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:23.768435    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/21c42a22-b64c-4528-bd63-d5d583ba59ab-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-hw4m5\" (UID: \"21c42a22-b64c-4528-bd63-d5d583ba59ab\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-hw4m5"
Sep 16 17:27:23.771120 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:23.771094    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/21c42a22-b64c-4528-bd63-d5d583ba59ab-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-hw4m5\" (UID: \"21c42a22-b64c-4528-bd63-d5d583ba59ab\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-hw4m5"
Sep 16 17:27:23.969985 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:23.969940    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-hw4m5"
Sep 16 17:27:24.252471 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:24.252438    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-hw4m5"]
Sep 16 17:27:24.255868 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:27:24.255818    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21c42a22_b64c_4528_bd63_d5d583ba59ab.slice/crio-da740a03a05dc6f890ea53460ad6a83af39e8b0eadfd097e731c1ab3e9daf370 WatchSource:0}: Error finding container da740a03a05dc6f890ea53460ad6a83af39e8b0eadfd097e731c1ab3e9daf370: Status 404 returned error can't find the container with id da740a03a05dc6f890ea53460ad6a83af39e8b0eadfd097e731c1ab3e9daf370
Sep 16 17:27:24.632376 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:24.632339    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-hw4m5" event={"ID":"21c42a22-b64c-4528-bd63-d5d583ba59ab","Type":"ContainerStarted","Data":"da740a03a05dc6f890ea53460ad6a83af39e8b0eadfd097e731c1ab3e9daf370"}
Sep 16 17:27:25.081258 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:25.081214    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3b4699f3-b8d0-408c-9031-37f1ec7cd316-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zhxvb\" (UID: \"3b4699f3-b8d0-408c-9031-37f1ec7cd316\") " pod="openshift-insights/insights-runtime-extractor-zhxvb"
Sep 16 17:27:25.084550 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:25.084519    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3b4699f3-b8d0-408c-9031-37f1ec7cd316-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zhxvb\" (UID: \"3b4699f3-b8d0-408c-9031-37f1ec7cd316\") " pod="openshift-insights/insights-runtime-extractor-zhxvb"
Sep 16 17:27:25.302784 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:25.302745    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-zhxvb"
Sep 16 17:27:25.733868 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:25.733825    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-zhxvb"]
Sep 16 17:27:25.736991 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:27:25.736960    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b4699f3_b8d0_408c_9031_37f1ec7cd316.slice/crio-2b7959d6e579cb9bb0c52bd9be9efdf2cb7bbf5e5f881bd8c3a5fe7bfd2e79aa WatchSource:0}: Error finding container 2b7959d6e579cb9bb0c52bd9be9efdf2cb7bbf5e5f881bd8c3a5fe7bfd2e79aa: Status 404 returned error can't find the container with id 2b7959d6e579cb9bb0c52bd9be9efdf2cb7bbf5e5f881bd8c3a5fe7bfd2e79aa
Sep 16 17:27:25.871044 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:27:25.870999    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-9cbqd" podUID="a639ab67-695c-4ec2-af20-1882ac49c4df"
Sep 16 17:27:26.640862 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:26.640822    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-zhxvb" event={"ID":"3b4699f3-b8d0-408c-9031-37f1ec7cd316","Type":"ContainerStarted","Data":"d50bc4643b7e0005626113ea9e3f2bc6e0374b8c0d01dff0841f6c48b0fcda73"}
Sep 16 17:27:26.641207 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:26.640902    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-zhxvb" event={"ID":"3b4699f3-b8d0-408c-9031-37f1ec7cd316","Type":"ContainerStarted","Data":"a4f4d5636781a6b3a7b29b6493ac4eff5fb13c70e82d69c83526679981bd4c5d"}
Sep 16 17:27:26.641207 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:26.640932    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-zhxvb" event={"ID":"3b4699f3-b8d0-408c-9031-37f1ec7cd316","Type":"ContainerStarted","Data":"2b7959d6e579cb9bb0c52bd9be9efdf2cb7bbf5e5f881bd8c3a5fe7bfd2e79aa"}
Sep 16 17:27:26.642252 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:26.642223    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-hw4m5" event={"ID":"21c42a22-b64c-4528-bd63-d5d583ba59ab","Type":"ContainerStarted","Data":"2a20f56a183aa681b156cd75e0825748c78bd46e77238a49648500f3ca3d1cc7"}
Sep 16 17:27:26.642352 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:26.642242    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-9cbqd"
Sep 16 17:27:26.642619 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:26.642596    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-hw4m5"
Sep 16 17:27:26.647127 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:26.647108    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-hw4m5"
Sep 16 17:27:26.752061 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:26.751999    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-hw4m5" podStartSLOduration=2.373961606 podStartE2EDuration="3.751973572s" totalImagesPullingTime="1.378011966s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:27:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:27:26.695165804 +0000 UTC m=+159.165163939" watchObservedRunningTime="2026-09-16 17:27:26.751973572 +0000 UTC m=+159.221971707"
Sep 16 17:27:27.121998 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:27:27.121956    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-xfgjq" podUID="bc719829-66b1-4a84-b994-e71cf111dafa"
Sep 16 17:27:28.017073 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:28.017032    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-9d65f485-4kk8t"]
Sep 16 17:27:28.019242 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:28.019216    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-9d65f485-4kk8t"
Sep 16 17:27:28.023379 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:28.023352    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 16 17:27:28.023964 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:28.023941    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 16 17:27:28.024067 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:28.023967    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 16 17:27:28.024067 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:28.024013    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 16 17:27:28.024169 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:28.024086    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 16 17:27:28.024522 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:28.024503    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-5pfdk\""
Sep 16 17:27:28.050591 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:28.050560    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-9d65f485-4kk8t"]
Sep 16 17:27:28.107150 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:28.107121    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9ba8e716-8f2b-4fe4-a140-707f7335fbef-console-oauth-config\") pod \"console-9d65f485-4kk8t\" (UID: \"9ba8e716-8f2b-4fe4-a140-707f7335fbef\") " pod="openshift-console/console-9d65f485-4kk8t"
Sep 16 17:27:28.107150 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:28.107154    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9ba8e716-8f2b-4fe4-a140-707f7335fbef-console-config\") pod \"console-9d65f485-4kk8t\" (UID: \"9ba8e716-8f2b-4fe4-a140-707f7335fbef\") " pod="openshift-console/console-9d65f485-4kk8t"
Sep 16 17:27:28.107438 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:28.107174    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9ba8e716-8f2b-4fe4-a140-707f7335fbef-console-serving-cert\") pod \"console-9d65f485-4kk8t\" (UID: \"9ba8e716-8f2b-4fe4-a140-707f7335fbef\") " pod="openshift-console/console-9d65f485-4kk8t"
Sep 16 17:27:28.107438 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:28.107231    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9ba8e716-8f2b-4fe4-a140-707f7335fbef-service-ca\") pod \"console-9d65f485-4kk8t\" (UID: \"9ba8e716-8f2b-4fe4-a140-707f7335fbef\") " pod="openshift-console/console-9d65f485-4kk8t"
Sep 16 17:27:28.107438 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:28.107341    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9ba8e716-8f2b-4fe4-a140-707f7335fbef-oauth-serving-cert\") pod \"console-9d65f485-4kk8t\" (UID: \"9ba8e716-8f2b-4fe4-a140-707f7335fbef\") " pod="openshift-console/console-9d65f485-4kk8t"
Sep 16 17:27:28.107438 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:28.107379    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zcl4r\" (UniqueName: \"kubernetes.io/projected/9ba8e716-8f2b-4fe4-a140-707f7335fbef-kube-api-access-zcl4r\") pod \"console-9d65f485-4kk8t\" (UID: \"9ba8e716-8f2b-4fe4-a140-707f7335fbef\") " pod="openshift-console/console-9d65f485-4kk8t"
Sep 16 17:27:28.208144 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:28.208050    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9ba8e716-8f2b-4fe4-a140-707f7335fbef-console-oauth-config\") pod \"console-9d65f485-4kk8t\" (UID: \"9ba8e716-8f2b-4fe4-a140-707f7335fbef\") " pod="openshift-console/console-9d65f485-4kk8t"
Sep 16 17:27:28.208144 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:28.208105    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9ba8e716-8f2b-4fe4-a140-707f7335fbef-console-config\") pod \"console-9d65f485-4kk8t\" (UID: \"9ba8e716-8f2b-4fe4-a140-707f7335fbef\") " pod="openshift-console/console-9d65f485-4kk8t"
Sep 16 17:27:28.208144 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:28.208136    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9ba8e716-8f2b-4fe4-a140-707f7335fbef-console-serving-cert\") pod \"console-9d65f485-4kk8t\" (UID: \"9ba8e716-8f2b-4fe4-a140-707f7335fbef\") " pod="openshift-console/console-9d65f485-4kk8t"
Sep 16 17:27:28.208425 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:28.208184    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9ba8e716-8f2b-4fe4-a140-707f7335fbef-service-ca\") pod \"console-9d65f485-4kk8t\" (UID: \"9ba8e716-8f2b-4fe4-a140-707f7335fbef\") " pod="openshift-console/console-9d65f485-4kk8t"
Sep 16 17:27:28.208425 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:28.208217    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9ba8e716-8f2b-4fe4-a140-707f7335fbef-oauth-serving-cert\") pod \"console-9d65f485-4kk8t\" (UID: \"9ba8e716-8f2b-4fe4-a140-707f7335fbef\") " pod="openshift-console/console-9d65f485-4kk8t"
Sep 16 17:27:28.208425 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:28.208242    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zcl4r\" (UniqueName: \"kubernetes.io/projected/9ba8e716-8f2b-4fe4-a140-707f7335fbef-kube-api-access-zcl4r\") pod \"console-9d65f485-4kk8t\" (UID: \"9ba8e716-8f2b-4fe4-a140-707f7335fbef\") " pod="openshift-console/console-9d65f485-4kk8t"
Sep 16 17:27:28.209043 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:28.208986    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9ba8e716-8f2b-4fe4-a140-707f7335fbef-console-config\") pod \"console-9d65f485-4kk8t\" (UID: \"9ba8e716-8f2b-4fe4-a140-707f7335fbef\") " pod="openshift-console/console-9d65f485-4kk8t"
Sep 16 17:27:28.209282 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:28.209259    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9ba8e716-8f2b-4fe4-a140-707f7335fbef-service-ca\") pod \"console-9d65f485-4kk8t\" (UID: \"9ba8e716-8f2b-4fe4-a140-707f7335fbef\") " pod="openshift-console/console-9d65f485-4kk8t"
Sep 16 17:27:28.209401 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:28.209290    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9ba8e716-8f2b-4fe4-a140-707f7335fbef-oauth-serving-cert\") pod \"console-9d65f485-4kk8t\" (UID: \"9ba8e716-8f2b-4fe4-a140-707f7335fbef\") " pod="openshift-console/console-9d65f485-4kk8t"
Sep 16 17:27:28.210917 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:28.210893    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9ba8e716-8f2b-4fe4-a140-707f7335fbef-console-oauth-config\") pod \"console-9d65f485-4kk8t\" (UID: \"9ba8e716-8f2b-4fe4-a140-707f7335fbef\") " pod="openshift-console/console-9d65f485-4kk8t"
Sep 16 17:27:28.211061 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:28.211038    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9ba8e716-8f2b-4fe4-a140-707f7335fbef-console-serving-cert\") pod \"console-9d65f485-4kk8t\" (UID: \"9ba8e716-8f2b-4fe4-a140-707f7335fbef\") " pod="openshift-console/console-9d65f485-4kk8t"
Sep 16 17:27:28.225066 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:28.225039    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zcl4r\" (UniqueName: \"kubernetes.io/projected/9ba8e716-8f2b-4fe4-a140-707f7335fbef-kube-api-access-zcl4r\") pod \"console-9d65f485-4kk8t\" (UID: \"9ba8e716-8f2b-4fe4-a140-707f7335fbef\") " pod="openshift-console/console-9d65f485-4kk8t"
Sep 16 17:27:28.330823 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:28.330786    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-9d65f485-4kk8t"
Sep 16 17:27:28.636675 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:28.636639    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-9d65f485-4kk8t"]
Sep 16 17:27:28.640232 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:27:28.640203    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ba8e716_8f2b_4fe4_a140_707f7335fbef.slice/crio-8d7b0c907df37d1911b750ef633997f43591ed03408ec15e336693a6bead0dc6 WatchSource:0}: Error finding container 8d7b0c907df37d1911b750ef633997f43591ed03408ec15e336693a6bead0dc6: Status 404 returned error can't find the container with id 8d7b0c907df37d1911b750ef633997f43591ed03408ec15e336693a6bead0dc6
Sep 16 17:27:28.653355 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:28.653324    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-zhxvb" event={"ID":"3b4699f3-b8d0-408c-9031-37f1ec7cd316","Type":"ContainerStarted","Data":"91964f525a9f5dda575e291adc83a90966be41f2b1d5cd8d68b6ed241b0d3c75"}
Sep 16 17:27:28.654880 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:28.654828    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-9d65f485-4kk8t" event={"ID":"9ba8e716-8f2b-4fe4-a140-707f7335fbef","Type":"ContainerStarted","Data":"8d7b0c907df37d1911b750ef633997f43591ed03408ec15e336693a6bead0dc6"}
Sep 16 17:27:30.730407 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:30.730369    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a639ab67-695c-4ec2-af20-1882ac49c4df-metrics-tls\") pod \"dns-default-9cbqd\" (UID: \"a639ab67-695c-4ec2-af20-1882ac49c4df\") " pod="openshift-dns/dns-default-9cbqd"
Sep 16 17:27:30.732983 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:30.732953    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a639ab67-695c-4ec2-af20-1882ac49c4df-metrics-tls\") pod \"dns-default-9cbqd\" (UID: \"a639ab67-695c-4ec2-af20-1882ac49c4df\") " pod="openshift-dns/dns-default-9cbqd"
Sep 16 17:27:30.846164 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:30.846127    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-v8tmk\""
Sep 16 17:27:30.853452 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:30.853424    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-9cbqd"
Sep 16 17:27:32.757991 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:32.757932    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-zhxvb" podStartSLOduration=37.802816935 podStartE2EDuration="39.757909115s" totalImagesPullingTime="1.95509218s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 17:26:53 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:27:28.688388238 +0000 UTC m=+161.158386374" watchObservedRunningTime="2026-09-16 17:27:32.757909115 +0000 UTC m=+165.227907250"
Sep 16 17:27:32.759105 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:32.759073    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-fbrpx"]
Sep 16 17:27:32.761544 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:32.761517    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-fbrpx"
Sep 16 17:27:32.766311 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:32.766289    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 16 17:27:32.770689 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:32.770671    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-rg679\""
Sep 16 17:27:32.770799 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:32.770712    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 16 17:27:32.789803 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:32.789769    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 16 17:27:32.801782 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:32.801748    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 16 17:27:32.847016 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:32.846979    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f8ebccfe-c8a8-4a90-8cad-c764e3ee9956-node-exporter-tls\") pod \"node-exporter-fbrpx\" (UID: \"f8ebccfe-c8a8-4a90-8cad-c764e3ee9956\") " pod="openshift-monitoring/node-exporter-fbrpx"
Sep 16 17:27:32.847172 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:32.847024    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f8ebccfe-c8a8-4a90-8cad-c764e3ee9956-node-exporter-wtmp\") pod \"node-exporter-fbrpx\" (UID: \"f8ebccfe-c8a8-4a90-8cad-c764e3ee9956\") " pod="openshift-monitoring/node-exporter-fbrpx"
Sep 16 17:27:32.847172 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:32.847094    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f8ebccfe-c8a8-4a90-8cad-c764e3ee9956-root\") pod \"node-exporter-fbrpx\" (UID: \"f8ebccfe-c8a8-4a90-8cad-c764e3ee9956\") " pod="openshift-monitoring/node-exporter-fbrpx"
Sep 16 17:27:32.847172 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:32.847144    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f8ebccfe-c8a8-4a90-8cad-c764e3ee9956-node-exporter-accelerators-collector-config\") pod \"node-exporter-fbrpx\" (UID: \"f8ebccfe-c8a8-4a90-8cad-c764e3ee9956\") " pod="openshift-monitoring/node-exporter-fbrpx"
Sep 16 17:27:32.847306 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:32.847187    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f8ebccfe-c8a8-4a90-8cad-c764e3ee9956-sys\") pod \"node-exporter-fbrpx\" (UID: \"f8ebccfe-c8a8-4a90-8cad-c764e3ee9956\") " pod="openshift-monitoring/node-exporter-fbrpx"
Sep 16 17:27:32.847306 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:32.847217    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-br4ln\" (UniqueName: \"kubernetes.io/projected/f8ebccfe-c8a8-4a90-8cad-c764e3ee9956-kube-api-access-br4ln\") pod \"node-exporter-fbrpx\" (UID: \"f8ebccfe-c8a8-4a90-8cad-c764e3ee9956\") " pod="openshift-monitoring/node-exporter-fbrpx"
Sep 16 17:27:32.847401 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:32.847322    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f8ebccfe-c8a8-4a90-8cad-c764e3ee9956-node-exporter-textfile\") pod \"node-exporter-fbrpx\" (UID: \"f8ebccfe-c8a8-4a90-8cad-c764e3ee9956\") " pod="openshift-monitoring/node-exporter-fbrpx"
Sep 16 17:27:32.847401 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:32.847351    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f8ebccfe-c8a8-4a90-8cad-c764e3ee9956-metrics-client-ca\") pod \"node-exporter-fbrpx\" (UID: \"f8ebccfe-c8a8-4a90-8cad-c764e3ee9956\") " pod="openshift-monitoring/node-exporter-fbrpx"
Sep 16 17:27:32.847401 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:32.847366    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f8ebccfe-c8a8-4a90-8cad-c764e3ee9956-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-fbrpx\" (UID: \"f8ebccfe-c8a8-4a90-8cad-c764e3ee9956\") " pod="openshift-monitoring/node-exporter-fbrpx"
Sep 16 17:27:32.948187 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:32.948145    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f8ebccfe-c8a8-4a90-8cad-c764e3ee9956-metrics-client-ca\") pod \"node-exporter-fbrpx\" (UID: \"f8ebccfe-c8a8-4a90-8cad-c764e3ee9956\") " pod="openshift-monitoring/node-exporter-fbrpx"
Sep 16 17:27:32.948187 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:32.948195    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f8ebccfe-c8a8-4a90-8cad-c764e3ee9956-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-fbrpx\" (UID: \"f8ebccfe-c8a8-4a90-8cad-c764e3ee9956\") " pod="openshift-monitoring/node-exporter-fbrpx"
Sep 16 17:27:32.948399 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:32.948217    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f8ebccfe-c8a8-4a90-8cad-c764e3ee9956-node-exporter-tls\") pod \"node-exporter-fbrpx\" (UID: \"f8ebccfe-c8a8-4a90-8cad-c764e3ee9956\") " pod="openshift-monitoring/node-exporter-fbrpx"
Sep 16 17:27:32.948399 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:32.948244    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f8ebccfe-c8a8-4a90-8cad-c764e3ee9956-node-exporter-wtmp\") pod \"node-exporter-fbrpx\" (UID: \"f8ebccfe-c8a8-4a90-8cad-c764e3ee9956\") " pod="openshift-monitoring/node-exporter-fbrpx"
Sep 16 17:27:32.948399 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:32.948270    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f8ebccfe-c8a8-4a90-8cad-c764e3ee9956-root\") pod \"node-exporter-fbrpx\" (UID: \"f8ebccfe-c8a8-4a90-8cad-c764e3ee9956\") " pod="openshift-monitoring/node-exporter-fbrpx"
Sep 16 17:27:32.948399 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:32.948319    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f8ebccfe-c8a8-4a90-8cad-c764e3ee9956-node-exporter-accelerators-collector-config\") pod \"node-exporter-fbrpx\" (UID: \"f8ebccfe-c8a8-4a90-8cad-c764e3ee9956\") " pod="openshift-monitoring/node-exporter-fbrpx"
Sep 16 17:27:32.948399 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:32.948346    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f8ebccfe-c8a8-4a90-8cad-c764e3ee9956-sys\") pod \"node-exporter-fbrpx\" (UID: \"f8ebccfe-c8a8-4a90-8cad-c764e3ee9956\") " pod="openshift-monitoring/node-exporter-fbrpx"
Sep 16 17:27:32.948399 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:32.948375    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-br4ln\" (UniqueName: \"kubernetes.io/projected/f8ebccfe-c8a8-4a90-8cad-c764e3ee9956-kube-api-access-br4ln\") pod \"node-exporter-fbrpx\" (UID: \"f8ebccfe-c8a8-4a90-8cad-c764e3ee9956\") " pod="openshift-monitoring/node-exporter-fbrpx"
Sep 16 17:27:32.948585 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:32.948461    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f8ebccfe-c8a8-4a90-8cad-c764e3ee9956-node-exporter-textfile\") pod \"node-exporter-fbrpx\" (UID: \"f8ebccfe-c8a8-4a90-8cad-c764e3ee9956\") " pod="openshift-monitoring/node-exporter-fbrpx"
Sep 16 17:27:32.948886 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:32.948714    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f8ebccfe-c8a8-4a90-8cad-c764e3ee9956-sys\") pod \"node-exporter-fbrpx\" (UID: \"f8ebccfe-c8a8-4a90-8cad-c764e3ee9956\") " pod="openshift-monitoring/node-exporter-fbrpx"
Sep 16 17:27:32.948886 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:32.948761    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f8ebccfe-c8a8-4a90-8cad-c764e3ee9956-node-exporter-textfile\") pod \"node-exporter-fbrpx\" (UID: \"f8ebccfe-c8a8-4a90-8cad-c764e3ee9956\") " pod="openshift-monitoring/node-exporter-fbrpx"
Sep 16 17:27:32.948886 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:32.948820    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f8ebccfe-c8a8-4a90-8cad-c764e3ee9956-metrics-client-ca\") pod \"node-exporter-fbrpx\" (UID: \"f8ebccfe-c8a8-4a90-8cad-c764e3ee9956\") " pod="openshift-monitoring/node-exporter-fbrpx"
Sep 16 17:27:32.948886 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:32.948881    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f8ebccfe-c8a8-4a90-8cad-c764e3ee9956-node-exporter-wtmp\") pod \"node-exporter-fbrpx\" (UID: \"f8ebccfe-c8a8-4a90-8cad-c764e3ee9956\") " pod="openshift-monitoring/node-exporter-fbrpx"
Sep 16 17:27:32.948886 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:32.948887    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f8ebccfe-c8a8-4a90-8cad-c764e3ee9956-root\") pod \"node-exporter-fbrpx\" (UID: \"f8ebccfe-c8a8-4a90-8cad-c764e3ee9956\") " pod="openshift-monitoring/node-exporter-fbrpx"
Sep 16 17:27:32.949163 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:32.949148    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f8ebccfe-c8a8-4a90-8cad-c764e3ee9956-node-exporter-accelerators-collector-config\") pod \"node-exporter-fbrpx\" (UID: \"f8ebccfe-c8a8-4a90-8cad-c764e3ee9956\") " pod="openshift-monitoring/node-exporter-fbrpx"
Sep 16 17:27:32.950904 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:32.950879    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f8ebccfe-c8a8-4a90-8cad-c764e3ee9956-node-exporter-tls\") pod \"node-exporter-fbrpx\" (UID: \"f8ebccfe-c8a8-4a90-8cad-c764e3ee9956\") " pod="openshift-monitoring/node-exporter-fbrpx"
Sep 16 17:27:32.951011 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:32.950989    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f8ebccfe-c8a8-4a90-8cad-c764e3ee9956-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-fbrpx\" (UID: \"f8ebccfe-c8a8-4a90-8cad-c764e3ee9956\") " pod="openshift-monitoring/node-exporter-fbrpx"
Sep 16 17:27:32.959945 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:32.959922    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-br4ln\" (UniqueName: \"kubernetes.io/projected/f8ebccfe-c8a8-4a90-8cad-c764e3ee9956-kube-api-access-br4ln\") pod \"node-exporter-fbrpx\" (UID: \"f8ebccfe-c8a8-4a90-8cad-c764e3ee9956\") " pod="openshift-monitoring/node-exporter-fbrpx"
Sep 16 17:27:33.072430 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:33.072396    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-fbrpx"
Sep 16 17:27:33.705710 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:33.705673    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 17:27:33.708933 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:33.708906    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 16 17:27:33.718869 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:33.718825    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 16 17:27:33.719411 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:33.719148    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 16 17:27:33.719411 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:33.719191    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 16 17:27:33.719411 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:33.719197    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 16 17:27:33.719411 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:33.719262    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 16 17:27:33.719411 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:33.719293    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 16 17:27:33.719411 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:33.719343    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 16 17:27:33.719411 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:33.719345    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 16 17:27:33.720391 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:33.720375    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-jkg9r\""
Sep 16 17:27:33.720391 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:33.720387    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 16 17:27:33.728993 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:33.728969    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 17:27:33.755679 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:33.755640    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/a2ad2862-5d15-46b2-aaeb-a96ad6af3255-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"a2ad2862-5d15-46b2-aaeb-a96ad6af3255\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 17:27:33.755895 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:33.755695    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a2ad2862-5d15-46b2-aaeb-a96ad6af3255-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"a2ad2862-5d15-46b2-aaeb-a96ad6af3255\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 17:27:33.755895 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:33.755795    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a2ad2862-5d15-46b2-aaeb-a96ad6af3255-config-out\") pod \"alertmanager-main-0\" (UID: \"a2ad2862-5d15-46b2-aaeb-a96ad6af3255\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 17:27:33.755895 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:33.755835    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a2ad2862-5d15-46b2-aaeb-a96ad6af3255-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"a2ad2862-5d15-46b2-aaeb-a96ad6af3255\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 17:27:33.756042 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:33.755920    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a2ad2862-5d15-46b2-aaeb-a96ad6af3255-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"a2ad2862-5d15-46b2-aaeb-a96ad6af3255\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 17:27:33.756042 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:33.755995    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a2ad2862-5d15-46b2-aaeb-a96ad6af3255-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"a2ad2862-5d15-46b2-aaeb-a96ad6af3255\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 17:27:33.756127 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:33.756047    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/a2ad2862-5d15-46b2-aaeb-a96ad6af3255-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"a2ad2862-5d15-46b2-aaeb-a96ad6af3255\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 17:27:33.756127 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:33.756101    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a2ad2862-5d15-46b2-aaeb-a96ad6af3255-tls-assets\") pod \"alertmanager-main-0\" (UID: \"a2ad2862-5d15-46b2-aaeb-a96ad6af3255\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 17:27:33.756228 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:33.756135    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a2ad2862-5d15-46b2-aaeb-a96ad6af3255-web-config\") pod \"alertmanager-main-0\" (UID: \"a2ad2862-5d15-46b2-aaeb-a96ad6af3255\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 17:27:33.756228 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:33.756175    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a2ad2862-5d15-46b2-aaeb-a96ad6af3255-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"a2ad2862-5d15-46b2-aaeb-a96ad6af3255\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 17:27:33.756228 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:33.756209    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a2ad2862-5d15-46b2-aaeb-a96ad6af3255-config-volume\") pod \"alertmanager-main-0\" (UID: \"a2ad2862-5d15-46b2-aaeb-a96ad6af3255\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 17:27:33.756362 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:33.756251    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-blprt\" (UniqueName: \"kubernetes.io/projected/a2ad2862-5d15-46b2-aaeb-a96ad6af3255-kube-api-access-blprt\") pod \"alertmanager-main-0\" (UID: \"a2ad2862-5d15-46b2-aaeb-a96ad6af3255\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 17:27:33.756362 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:33.756285    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a2ad2862-5d15-46b2-aaeb-a96ad6af3255-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"a2ad2862-5d15-46b2-aaeb-a96ad6af3255\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 17:27:33.857287 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:33.857242    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a2ad2862-5d15-46b2-aaeb-a96ad6af3255-config-volume\") pod \"alertmanager-main-0\" (UID: \"a2ad2862-5d15-46b2-aaeb-a96ad6af3255\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 17:27:33.857778 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:33.857300    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blprt\" (UniqueName: \"kubernetes.io/projected/a2ad2862-5d15-46b2-aaeb-a96ad6af3255-kube-api-access-blprt\") pod \"alertmanager-main-0\" (UID: \"a2ad2862-5d15-46b2-aaeb-a96ad6af3255\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 17:27:33.857778 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:33.857327    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a2ad2862-5d15-46b2-aaeb-a96ad6af3255-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"a2ad2862-5d15-46b2-aaeb-a96ad6af3255\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 17:27:33.857778 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:33.857359    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/a2ad2862-5d15-46b2-aaeb-a96ad6af3255-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"a2ad2862-5d15-46b2-aaeb-a96ad6af3255\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 17:27:33.857778 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:33.857389    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a2ad2862-5d15-46b2-aaeb-a96ad6af3255-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"a2ad2862-5d15-46b2-aaeb-a96ad6af3255\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 17:27:33.857778 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:33.857430    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a2ad2862-5d15-46b2-aaeb-a96ad6af3255-config-out\") pod \"alertmanager-main-0\" (UID: \"a2ad2862-5d15-46b2-aaeb-a96ad6af3255\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 17:27:33.857778 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:33.857455    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a2ad2862-5d15-46b2-aaeb-a96ad6af3255-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"a2ad2862-5d15-46b2-aaeb-a96ad6af3255\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 17:27:33.857778 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:33.857489    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a2ad2862-5d15-46b2-aaeb-a96ad6af3255-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"a2ad2862-5d15-46b2-aaeb-a96ad6af3255\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 17:27:33.857778 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:33.857513    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a2ad2862-5d15-46b2-aaeb-a96ad6af3255-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"a2ad2862-5d15-46b2-aaeb-a96ad6af3255\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 17:27:33.857778 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:33.857550    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/a2ad2862-5d15-46b2-aaeb-a96ad6af3255-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"a2ad2862-5d15-46b2-aaeb-a96ad6af3255\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 17:27:33.857778 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:33.857586    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a2ad2862-5d15-46b2-aaeb-a96ad6af3255-tls-assets\") pod \"alertmanager-main-0\" (UID: \"a2ad2862-5d15-46b2-aaeb-a96ad6af3255\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 17:27:33.857778 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:33.857627    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a2ad2862-5d15-46b2-aaeb-a96ad6af3255-web-config\") pod \"alertmanager-main-0\" (UID: \"a2ad2862-5d15-46b2-aaeb-a96ad6af3255\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 17:27:33.857778 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:33.857656    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a2ad2862-5d15-46b2-aaeb-a96ad6af3255-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"a2ad2862-5d15-46b2-aaeb-a96ad6af3255\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 17:27:33.858402 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:27:33.857807    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a2ad2862-5d15-46b2-aaeb-a96ad6af3255-alertmanager-trusted-ca-bundle podName:a2ad2862-5d15-46b2-aaeb-a96ad6af3255 nodeName:}" failed. No retries permitted until 2026-09-16 17:27:34.357787375 +0000 UTC m=+166.827785492 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "alertmanager-trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/a2ad2862-5d15-46b2-aaeb-a96ad6af3255-alertmanager-trusted-ca-bundle") pod "alertmanager-main-0" (UID: "a2ad2862-5d15-46b2-aaeb-a96ad6af3255") : configmap references non-existent config key: ca-bundle.crt
Sep 16 17:27:33.858402 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:33.858304    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/a2ad2862-5d15-46b2-aaeb-a96ad6af3255-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"a2ad2862-5d15-46b2-aaeb-a96ad6af3255\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 17:27:33.858402 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:33.858397    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a2ad2862-5d15-46b2-aaeb-a96ad6af3255-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"a2ad2862-5d15-46b2-aaeb-a96ad6af3255\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 17:27:33.858572 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:27:33.858499    2579 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found
Sep 16 17:27:33.858572 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:27:33.858562    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a2ad2862-5d15-46b2-aaeb-a96ad6af3255-secret-alertmanager-main-tls podName:a2ad2862-5d15-46b2-aaeb-a96ad6af3255 nodeName:}" failed. No retries permitted until 2026-09-16 17:27:34.358544808 +0000 UTC m=+166.828542927 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/a2ad2862-5d15-46b2-aaeb-a96ad6af3255-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "a2ad2862-5d15-46b2-aaeb-a96ad6af3255") : secret "alertmanager-main-tls" not found
Sep 16 17:27:33.860764 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:33.860707    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a2ad2862-5d15-46b2-aaeb-a96ad6af3255-config-volume\") pod \"alertmanager-main-0\" (UID: \"a2ad2862-5d15-46b2-aaeb-a96ad6af3255\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 17:27:33.861697 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:33.861665    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a2ad2862-5d15-46b2-aaeb-a96ad6af3255-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"a2ad2862-5d15-46b2-aaeb-a96ad6af3255\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 17:27:33.862569 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:33.862468    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a2ad2862-5d15-46b2-aaeb-a96ad6af3255-web-config\") pod \"alertmanager-main-0\" (UID: \"a2ad2862-5d15-46b2-aaeb-a96ad6af3255\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 17:27:33.862758 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:33.862711    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a2ad2862-5d15-46b2-aaeb-a96ad6af3255-tls-assets\") pod \"alertmanager-main-0\" (UID: \"a2ad2862-5d15-46b2-aaeb-a96ad6af3255\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 17:27:33.862984 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:33.862964    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a2ad2862-5d15-46b2-aaeb-a96ad6af3255-config-out\") pod \"alertmanager-main-0\" (UID: \"a2ad2862-5d15-46b2-aaeb-a96ad6af3255\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 17:27:33.863267 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:33.863249    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/a2ad2862-5d15-46b2-aaeb-a96ad6af3255-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"a2ad2862-5d15-46b2-aaeb-a96ad6af3255\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 17:27:33.863324 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:33.863273    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a2ad2862-5d15-46b2-aaeb-a96ad6af3255-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"a2ad2862-5d15-46b2-aaeb-a96ad6af3255\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 17:27:33.864622 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:33.864598    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a2ad2862-5d15-46b2-aaeb-a96ad6af3255-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"a2ad2862-5d15-46b2-aaeb-a96ad6af3255\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 17:27:33.869550 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:33.869530    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-blprt\" (UniqueName: \"kubernetes.io/projected/a2ad2862-5d15-46b2-aaeb-a96ad6af3255-kube-api-access-blprt\") pod \"alertmanager-main-0\" (UID: \"a2ad2862-5d15-46b2-aaeb-a96ad6af3255\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 17:27:34.333638 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:34.333597    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-d77b4b4b-2f2bv"]
Sep 16 17:27:34.337067 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:34.337044    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-d77b4b4b-2f2bv"
Sep 16 17:27:34.346109 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:34.345713    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 16 17:27:34.363561 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:34.363515    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a2ad2862-5d15-46b2-aaeb-a96ad6af3255-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"a2ad2862-5d15-46b2-aaeb-a96ad6af3255\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 17:27:34.363727 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:34.363568    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a2ad2862-5d15-46b2-aaeb-a96ad6af3255-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"a2ad2862-5d15-46b2-aaeb-a96ad6af3255\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 17:27:34.364616 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:34.364583    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a2ad2862-5d15-46b2-aaeb-a96ad6af3255-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"a2ad2862-5d15-46b2-aaeb-a96ad6af3255\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 17:27:34.366424 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:34.366394    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a2ad2862-5d15-46b2-aaeb-a96ad6af3255-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"a2ad2862-5d15-46b2-aaeb-a96ad6af3255\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 17:27:34.396485 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:34.396440    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-d77b4b4b-2f2bv"]
Sep 16 17:27:34.464925 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:34.464890    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3c32493a-a573-4c14-9e85-9f0faef07cd5-trusted-ca-bundle\") pod \"console-d77b4b4b-2f2bv\" (UID: \"3c32493a-a573-4c14-9e85-9f0faef07cd5\") " pod="openshift-console/console-d77b4b4b-2f2bv"
Sep 16 17:27:34.465097 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:34.464945    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3c32493a-a573-4c14-9e85-9f0faef07cd5-service-ca\") pod \"console-d77b4b4b-2f2bv\" (UID: \"3c32493a-a573-4c14-9e85-9f0faef07cd5\") " pod="openshift-console/console-d77b4b4b-2f2bv"
Sep 16 17:27:34.465097 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:34.464970    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3c32493a-a573-4c14-9e85-9f0faef07cd5-oauth-serving-cert\") pod \"console-d77b4b4b-2f2bv\" (UID: \"3c32493a-a573-4c14-9e85-9f0faef07cd5\") " pod="openshift-console/console-d77b4b4b-2f2bv"
Sep 16 17:27:34.465097 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:34.465046    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3c32493a-a573-4c14-9e85-9f0faef07cd5-console-serving-cert\") pod \"console-d77b4b4b-2f2bv\" (UID: \"3c32493a-a573-4c14-9e85-9f0faef07cd5\") " pod="openshift-console/console-d77b4b4b-2f2bv"
Sep 16 17:27:34.465097 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:34.465071    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgh8f\" (UniqueName: \"kubernetes.io/projected/3c32493a-a573-4c14-9e85-9f0faef07cd5-kube-api-access-zgh8f\") pod \"console-d77b4b4b-2f2bv\" (UID: \"3c32493a-a573-4c14-9e85-9f0faef07cd5\") " pod="openshift-console/console-d77b4b4b-2f2bv"
Sep 16 17:27:34.465277 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:34.465097    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3c32493a-a573-4c14-9e85-9f0faef07cd5-console-oauth-config\") pod \"console-d77b4b4b-2f2bv\" (UID: \"3c32493a-a573-4c14-9e85-9f0faef07cd5\") " pod="openshift-console/console-d77b4b4b-2f2bv"
Sep 16 17:27:34.465277 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:34.465128    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3c32493a-a573-4c14-9e85-9f0faef07cd5-console-config\") pod \"console-d77b4b4b-2f2bv\" (UID: \"3c32493a-a573-4c14-9e85-9f0faef07cd5\") " pod="openshift-console/console-d77b4b4b-2f2bv"
Sep 16 17:27:34.565784 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:34.565746    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3c32493a-a573-4c14-9e85-9f0faef07cd5-service-ca\") pod \"console-d77b4b4b-2f2bv\" (UID: \"3c32493a-a573-4c14-9e85-9f0faef07cd5\") " pod="openshift-console/console-d77b4b4b-2f2bv"
Sep 16 17:27:34.565784 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:34.565790    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3c32493a-a573-4c14-9e85-9f0faef07cd5-oauth-serving-cert\") pod \"console-d77b4b4b-2f2bv\" (UID: \"3c32493a-a573-4c14-9e85-9f0faef07cd5\") " pod="openshift-console/console-d77b4b4b-2f2bv"
Sep 16 17:27:34.566060 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:34.565834    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3c32493a-a573-4c14-9e85-9f0faef07cd5-console-serving-cert\") pod \"console-d77b4b4b-2f2bv\" (UID: \"3c32493a-a573-4c14-9e85-9f0faef07cd5\") " pod="openshift-console/console-d77b4b4b-2f2bv"
Sep 16 17:27:34.566060 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:34.565875    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgh8f\" (UniqueName: \"kubernetes.io/projected/3c32493a-a573-4c14-9e85-9f0faef07cd5-kube-api-access-zgh8f\") pod \"console-d77b4b4b-2f2bv\" (UID: \"3c32493a-a573-4c14-9e85-9f0faef07cd5\") " pod="openshift-console/console-d77b4b4b-2f2bv"
Sep 16 17:27:34.566060 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:34.565898    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3c32493a-a573-4c14-9e85-9f0faef07cd5-console-oauth-config\") pod \"console-d77b4b4b-2f2bv\" (UID: \"3c32493a-a573-4c14-9e85-9f0faef07cd5\") " pod="openshift-console/console-d77b4b4b-2f2bv"
Sep 16 17:27:34.566060 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:34.565958    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3c32493a-a573-4c14-9e85-9f0faef07cd5-console-config\") pod \"console-d77b4b4b-2f2bv\" (UID: \"3c32493a-a573-4c14-9e85-9f0faef07cd5\") " pod="openshift-console/console-d77b4b4b-2f2bv"
Sep 16 17:27:34.566060 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:34.566036    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3c32493a-a573-4c14-9e85-9f0faef07cd5-trusted-ca-bundle\") pod \"console-d77b4b4b-2f2bv\" (UID: \"3c32493a-a573-4c14-9e85-9f0faef07cd5\") " pod="openshift-console/console-d77b4b4b-2f2bv"
Sep 16 17:27:34.566653 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:34.566599    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3c32493a-a573-4c14-9e85-9f0faef07cd5-service-ca\") pod \"console-d77b4b4b-2f2bv\" (UID: \"3c32493a-a573-4c14-9e85-9f0faef07cd5\") " pod="openshift-console/console-d77b4b4b-2f2bv"
Sep 16 17:27:34.566653 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:34.566632    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3c32493a-a573-4c14-9e85-9f0faef07cd5-oauth-serving-cert\") pod \"console-d77b4b4b-2f2bv\" (UID: \"3c32493a-a573-4c14-9e85-9f0faef07cd5\") " pod="openshift-console/console-d77b4b4b-2f2bv"
Sep 16 17:27:34.567123 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:34.567099    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3c32493a-a573-4c14-9e85-9f0faef07cd5-trusted-ca-bundle\") pod \"console-d77b4b4b-2f2bv\" (UID: \"3c32493a-a573-4c14-9e85-9f0faef07cd5\") " pod="openshift-console/console-d77b4b4b-2f2bv"
Sep 16 17:27:34.567594 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:34.567570    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3c32493a-a573-4c14-9e85-9f0faef07cd5-console-config\") pod \"console-d77b4b4b-2f2bv\" (UID: \"3c32493a-a573-4c14-9e85-9f0faef07cd5\") " pod="openshift-console/console-d77b4b4b-2f2bv"
Sep 16 17:27:34.568599 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:34.568575    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3c32493a-a573-4c14-9e85-9f0faef07cd5-console-oauth-config\") pod \"console-d77b4b4b-2f2bv\" (UID: \"3c32493a-a573-4c14-9e85-9f0faef07cd5\") " pod="openshift-console/console-d77b4b4b-2f2bv"
Sep 16 17:27:34.569107 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:34.569083    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3c32493a-a573-4c14-9e85-9f0faef07cd5-console-serving-cert\") pod \"console-d77b4b4b-2f2bv\" (UID: \"3c32493a-a573-4c14-9e85-9f0faef07cd5\") " pod="openshift-console/console-d77b4b4b-2f2bv"
Sep 16 17:27:34.583904 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:34.583810    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgh8f\" (UniqueName: \"kubernetes.io/projected/3c32493a-a573-4c14-9e85-9f0faef07cd5-kube-api-access-zgh8f\") pod \"console-d77b4b4b-2f2bv\" (UID: \"3c32493a-a573-4c14-9e85-9f0faef07cd5\") " pod="openshift-console/console-d77b4b4b-2f2bv"
Sep 16 17:27:34.620718 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:34.620682    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 16 17:27:34.649436 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:34.649376    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-d77b4b4b-2f2bv"
Sep 16 17:27:35.901125 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:35.901085    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-654695c99f-4jmw9"]
Sep 16 17:27:35.904084 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:35.904061    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-654695c99f-4jmw9"
Sep 16 17:27:35.907541 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:35.907519    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Sep 16 17:27:35.907646 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:35.907580    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-f844si82u6nf9\""
Sep 16 17:27:35.907703 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:35.907657    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Sep 16 17:27:35.908054 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:35.908035    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Sep 16 17:27:35.908170 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:35.908098    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Sep 16 17:27:35.908323 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:35.908303    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-gmsqt\""
Sep 16 17:27:35.908635 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:35.908617    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Sep 16 17:27:35.925969 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:35.925942    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-654695c99f-4jmw9"]
Sep 16 17:27:35.980577 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:35.980534    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/27605ac2-4ce2-4847-b391-7f803ed3f809-metrics-client-ca\") pod \"thanos-querier-654695c99f-4jmw9\" (UID: \"27605ac2-4ce2-4847-b391-7f803ed3f809\") " pod="openshift-monitoring/thanos-querier-654695c99f-4jmw9"
Sep 16 17:27:35.980739 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:35.980584    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/27605ac2-4ce2-4847-b391-7f803ed3f809-secret-thanos-querier-tls\") pod \"thanos-querier-654695c99f-4jmw9\" (UID: \"27605ac2-4ce2-4847-b391-7f803ed3f809\") " pod="openshift-monitoring/thanos-querier-654695c99f-4jmw9"
Sep 16 17:27:35.980739 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:35.980610    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/27605ac2-4ce2-4847-b391-7f803ed3f809-secret-grpc-tls\") pod \"thanos-querier-654695c99f-4jmw9\" (UID: \"27605ac2-4ce2-4847-b391-7f803ed3f809\") " pod="openshift-monitoring/thanos-querier-654695c99f-4jmw9"
Sep 16 17:27:35.980739 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:35.980640    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/27605ac2-4ce2-4847-b391-7f803ed3f809-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-654695c99f-4jmw9\" (UID: \"27605ac2-4ce2-4847-b391-7f803ed3f809\") " pod="openshift-monitoring/thanos-querier-654695c99f-4jmw9"
Sep 16 17:27:35.980739 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:35.980679    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/27605ac2-4ce2-4847-b391-7f803ed3f809-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-654695c99f-4jmw9\" (UID: \"27605ac2-4ce2-4847-b391-7f803ed3f809\") " pod="openshift-monitoring/thanos-querier-654695c99f-4jmw9"
Sep 16 17:27:35.980966 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:35.980747    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/27605ac2-4ce2-4847-b391-7f803ed3f809-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-654695c99f-4jmw9\" (UID: \"27605ac2-4ce2-4847-b391-7f803ed3f809\") " pod="openshift-monitoring/thanos-querier-654695c99f-4jmw9"
Sep 16 17:27:35.980966 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:35.980804    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/27605ac2-4ce2-4847-b391-7f803ed3f809-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-654695c99f-4jmw9\" (UID: \"27605ac2-4ce2-4847-b391-7f803ed3f809\") " pod="openshift-monitoring/thanos-querier-654695c99f-4jmw9"
Sep 16 17:27:35.980966 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:35.980834    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p58fr\" (UniqueName: \"kubernetes.io/projected/27605ac2-4ce2-4847-b391-7f803ed3f809-kube-api-access-p58fr\") pod \"thanos-querier-654695c99f-4jmw9\" (UID: \"27605ac2-4ce2-4847-b391-7f803ed3f809\") " pod="openshift-monitoring/thanos-querier-654695c99f-4jmw9"
Sep 16 17:27:36.081834 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:36.081791    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/27605ac2-4ce2-4847-b391-7f803ed3f809-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-654695c99f-4jmw9\" (UID: \"27605ac2-4ce2-4847-b391-7f803ed3f809\") " pod="openshift-monitoring/thanos-querier-654695c99f-4jmw9"
Sep 16 17:27:36.082018 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:36.081840    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/27605ac2-4ce2-4847-b391-7f803ed3f809-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-654695c99f-4jmw9\" (UID: \"27605ac2-4ce2-4847-b391-7f803ed3f809\") " pod="openshift-monitoring/thanos-querier-654695c99f-4jmw9"
Sep 16 17:27:36.082095 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:36.082049    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p58fr\" (UniqueName: \"kubernetes.io/projected/27605ac2-4ce2-4847-b391-7f803ed3f809-kube-api-access-p58fr\") pod \"thanos-querier-654695c99f-4jmw9\" (UID: \"27605ac2-4ce2-4847-b391-7f803ed3f809\") " pod="openshift-monitoring/thanos-querier-654695c99f-4jmw9"
Sep 16 17:27:36.082208 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:36.082189    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/27605ac2-4ce2-4847-b391-7f803ed3f809-metrics-client-ca\") pod \"thanos-querier-654695c99f-4jmw9\" (UID: \"27605ac2-4ce2-4847-b391-7f803ed3f809\") " pod="openshift-monitoring/thanos-querier-654695c99f-4jmw9"
Sep 16 17:27:36.082290 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:36.082232    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/27605ac2-4ce2-4847-b391-7f803ed3f809-secret-thanos-querier-tls\") pod \"thanos-querier-654695c99f-4jmw9\" (UID: \"27605ac2-4ce2-4847-b391-7f803ed3f809\") " pod="openshift-monitoring/thanos-querier-654695c99f-4jmw9"
Sep 16 17:27:36.082290 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:36.082257    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/27605ac2-4ce2-4847-b391-7f803ed3f809-secret-grpc-tls\") pod \"thanos-querier-654695c99f-4jmw9\" (UID: \"27605ac2-4ce2-4847-b391-7f803ed3f809\") " pod="openshift-monitoring/thanos-querier-654695c99f-4jmw9"
Sep 16 17:27:36.082392 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:36.082287    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/27605ac2-4ce2-4847-b391-7f803ed3f809-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-654695c99f-4jmw9\" (UID: \"27605ac2-4ce2-4847-b391-7f803ed3f809\") " pod="openshift-monitoring/thanos-querier-654695c99f-4jmw9"
Sep 16 17:27:36.082392 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:36.082330    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/27605ac2-4ce2-4847-b391-7f803ed3f809-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-654695c99f-4jmw9\" (UID: \"27605ac2-4ce2-4847-b391-7f803ed3f809\") " pod="openshift-monitoring/thanos-querier-654695c99f-4jmw9"
Sep 16 17:27:36.083059 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:36.083007    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/27605ac2-4ce2-4847-b391-7f803ed3f809-metrics-client-ca\") pod \"thanos-querier-654695c99f-4jmw9\" (UID: \"27605ac2-4ce2-4847-b391-7f803ed3f809\") " pod="openshift-monitoring/thanos-querier-654695c99f-4jmw9"
Sep 16 17:27:36.085032 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:36.085000    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/27605ac2-4ce2-4847-b391-7f803ed3f809-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-654695c99f-4jmw9\" (UID: \"27605ac2-4ce2-4847-b391-7f803ed3f809\") " pod="openshift-monitoring/thanos-querier-654695c99f-4jmw9"
Sep 16 17:27:36.085316 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:36.085270    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/27605ac2-4ce2-4847-b391-7f803ed3f809-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-654695c99f-4jmw9\" (UID: \"27605ac2-4ce2-4847-b391-7f803ed3f809\") " pod="openshift-monitoring/thanos-querier-654695c99f-4jmw9"
Sep 16 17:27:36.085420 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:36.085377    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/27605ac2-4ce2-4847-b391-7f803ed3f809-secret-thanos-querier-tls\") pod \"thanos-querier-654695c99f-4jmw9\" (UID: \"27605ac2-4ce2-4847-b391-7f803ed3f809\") " pod="openshift-monitoring/thanos-querier-654695c99f-4jmw9"
Sep 16 17:27:36.085485 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:36.085433    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/27605ac2-4ce2-4847-b391-7f803ed3f809-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-654695c99f-4jmw9\" (UID: \"27605ac2-4ce2-4847-b391-7f803ed3f809\") " pod="openshift-monitoring/thanos-querier-654695c99f-4jmw9"
Sep 16 17:27:36.085812 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:36.085789    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/27605ac2-4ce2-4847-b391-7f803ed3f809-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-654695c99f-4jmw9\" (UID: \"27605ac2-4ce2-4847-b391-7f803ed3f809\") " pod="openshift-monitoring/thanos-querier-654695c99f-4jmw9"
Sep 16 17:27:36.086344 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:36.086326    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/27605ac2-4ce2-4847-b391-7f803ed3f809-secret-grpc-tls\") pod \"thanos-querier-654695c99f-4jmw9\" (UID: \"27605ac2-4ce2-4847-b391-7f803ed3f809\") " pod="openshift-monitoring/thanos-querier-654695c99f-4jmw9"
Sep 16 17:27:36.090698 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:36.090677    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p58fr\" (UniqueName: \"kubernetes.io/projected/27605ac2-4ce2-4847-b391-7f803ed3f809-kube-api-access-p58fr\") pod \"thanos-querier-654695c99f-4jmw9\" (UID: \"27605ac2-4ce2-4847-b391-7f803ed3f809\") " pod="openshift-monitoring/thanos-querier-654695c99f-4jmw9"
Sep 16 17:27:36.214946 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:36.214859    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-654695c99f-4jmw9"
Sep 16 17:27:37.128557 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:37.128524    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-5b8b89bf95-slsg6"]
Sep 16 17:27:37.133617 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:37.133591    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-5b8b89bf95-slsg6"
Sep 16 17:27:37.136308 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:37.136284    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 16 17:27:37.136445 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:37.136359    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Sep 16 17:27:37.137267 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:37.137151    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-dthdd\""
Sep 16 17:27:37.137267 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:37.137191    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Sep 16 17:27:37.137267 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:37.137151    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Sep 16 17:27:37.137267 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:37.137156    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dasi4duqqd8a2\""
Sep 16 17:27:37.141560 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:37.141536    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-5b8b89bf95-slsg6"]
Sep 16 17:27:37.193201 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:37.193159    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/2a7ca0b4-ec03-407e-bade-0a533e7c1b15-secret-metrics-server-client-certs\") pod \"metrics-server-5b8b89bf95-slsg6\" (UID: \"2a7ca0b4-ec03-407e-bade-0a533e7c1b15\") " pod="openshift-monitoring/metrics-server-5b8b89bf95-slsg6"
Sep 16 17:27:37.193440 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:37.193217    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2a7ca0b4-ec03-407e-bade-0a533e7c1b15-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5b8b89bf95-slsg6\" (UID: \"2a7ca0b4-ec03-407e-bade-0a533e7c1b15\") " pod="openshift-monitoring/metrics-server-5b8b89bf95-slsg6"
Sep 16 17:27:37.193440 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:37.193240    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/2a7ca0b4-ec03-407e-bade-0a533e7c1b15-secret-metrics-server-tls\") pod \"metrics-server-5b8b89bf95-slsg6\" (UID: \"2a7ca0b4-ec03-407e-bade-0a533e7c1b15\") " pod="openshift-monitoring/metrics-server-5b8b89bf95-slsg6"
Sep 16 17:27:37.193440 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:37.193259    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/2a7ca0b4-ec03-407e-bade-0a533e7c1b15-metrics-server-audit-profiles\") pod \"metrics-server-5b8b89bf95-slsg6\" (UID: \"2a7ca0b4-ec03-407e-bade-0a533e7c1b15\") " pod="openshift-monitoring/metrics-server-5b8b89bf95-slsg6"
Sep 16 17:27:37.193440 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:37.193313    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vw8nj\" (UniqueName: \"kubernetes.io/projected/2a7ca0b4-ec03-407e-bade-0a533e7c1b15-kube-api-access-vw8nj\") pod \"metrics-server-5b8b89bf95-slsg6\" (UID: \"2a7ca0b4-ec03-407e-bade-0a533e7c1b15\") " pod="openshift-monitoring/metrics-server-5b8b89bf95-slsg6"
Sep 16 17:27:37.193440 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:37.193336    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/2a7ca0b4-ec03-407e-bade-0a533e7c1b15-audit-log\") pod \"metrics-server-5b8b89bf95-slsg6\" (UID: \"2a7ca0b4-ec03-407e-bade-0a533e7c1b15\") " pod="openshift-monitoring/metrics-server-5b8b89bf95-slsg6"
Sep 16 17:27:37.193440 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:37.193388    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/2a7ca0b4-ec03-407e-bade-0a533e7c1b15-client-ca-bundle\") pod \"metrics-server-5b8b89bf95-slsg6\" (UID: \"2a7ca0b4-ec03-407e-bade-0a533e7c1b15\") " pod="openshift-monitoring/metrics-server-5b8b89bf95-slsg6"
Sep 16 17:27:37.294279 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:37.294234    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/2a7ca0b4-ec03-407e-bade-0a533e7c1b15-secret-metrics-server-client-certs\") pod \"metrics-server-5b8b89bf95-slsg6\" (UID: \"2a7ca0b4-ec03-407e-bade-0a533e7c1b15\") " pod="openshift-monitoring/metrics-server-5b8b89bf95-slsg6"
Sep 16 17:27:37.294279 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:37.294281    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2a7ca0b4-ec03-407e-bade-0a533e7c1b15-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5b8b89bf95-slsg6\" (UID: \"2a7ca0b4-ec03-407e-bade-0a533e7c1b15\") " pod="openshift-monitoring/metrics-server-5b8b89bf95-slsg6"
Sep 16 17:27:37.294506 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:37.294304    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/2a7ca0b4-ec03-407e-bade-0a533e7c1b15-secret-metrics-server-tls\") pod \"metrics-server-5b8b89bf95-slsg6\" (UID: \"2a7ca0b4-ec03-407e-bade-0a533e7c1b15\") " pod="openshift-monitoring/metrics-server-5b8b89bf95-slsg6"
Sep 16 17:27:37.294506 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:37.294329    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/2a7ca0b4-ec03-407e-bade-0a533e7c1b15-metrics-server-audit-profiles\") pod \"metrics-server-5b8b89bf95-slsg6\" (UID: \"2a7ca0b4-ec03-407e-bade-0a533e7c1b15\") " pod="openshift-monitoring/metrics-server-5b8b89bf95-slsg6"
Sep 16 17:27:37.294506 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:37.294379    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vw8nj\" (UniqueName: \"kubernetes.io/projected/2a7ca0b4-ec03-407e-bade-0a533e7c1b15-kube-api-access-vw8nj\") pod \"metrics-server-5b8b89bf95-slsg6\" (UID: \"2a7ca0b4-ec03-407e-bade-0a533e7c1b15\") " pod="openshift-monitoring/metrics-server-5b8b89bf95-slsg6"
Sep 16 17:27:37.294506 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:37.294405    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/2a7ca0b4-ec03-407e-bade-0a533e7c1b15-audit-log\") pod \"metrics-server-5b8b89bf95-slsg6\" (UID: \"2a7ca0b4-ec03-407e-bade-0a533e7c1b15\") " pod="openshift-monitoring/metrics-server-5b8b89bf95-slsg6"
Sep 16 17:27:37.294506 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:37.294445    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/2a7ca0b4-ec03-407e-bade-0a533e7c1b15-client-ca-bundle\") pod \"metrics-server-5b8b89bf95-slsg6\" (UID: \"2a7ca0b4-ec03-407e-bade-0a533e7c1b15\") " pod="openshift-monitoring/metrics-server-5b8b89bf95-slsg6"
Sep 16 17:27:37.296111 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:37.295436    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/2a7ca0b4-ec03-407e-bade-0a533e7c1b15-audit-log\") pod \"metrics-server-5b8b89bf95-slsg6\" (UID: \"2a7ca0b4-ec03-407e-bade-0a533e7c1b15\") " pod="openshift-monitoring/metrics-server-5b8b89bf95-slsg6"
Sep 16 17:27:37.296438 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:37.296412    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2a7ca0b4-ec03-407e-bade-0a533e7c1b15-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5b8b89bf95-slsg6\" (UID: \"2a7ca0b4-ec03-407e-bade-0a533e7c1b15\") " pod="openshift-monitoring/metrics-server-5b8b89bf95-slsg6"
Sep 16 17:27:37.297340 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:37.297233    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/2a7ca0b4-ec03-407e-bade-0a533e7c1b15-metrics-server-audit-profiles\") pod \"metrics-server-5b8b89bf95-slsg6\" (UID: \"2a7ca0b4-ec03-407e-bade-0a533e7c1b15\") " pod="openshift-monitoring/metrics-server-5b8b89bf95-slsg6"
Sep 16 17:27:37.298284 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:37.298228    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/2a7ca0b4-ec03-407e-bade-0a533e7c1b15-client-ca-bundle\") pod \"metrics-server-5b8b89bf95-slsg6\" (UID: \"2a7ca0b4-ec03-407e-bade-0a533e7c1b15\") " pod="openshift-monitoring/metrics-server-5b8b89bf95-slsg6"
Sep 16 17:27:37.298620 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:37.298564    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/2a7ca0b4-ec03-407e-bade-0a533e7c1b15-secret-metrics-server-client-certs\") pod \"metrics-server-5b8b89bf95-slsg6\" (UID: \"2a7ca0b4-ec03-407e-bade-0a533e7c1b15\") " pod="openshift-monitoring/metrics-server-5b8b89bf95-slsg6"
Sep 16 17:27:37.299691 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:37.299670    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/2a7ca0b4-ec03-407e-bade-0a533e7c1b15-secret-metrics-server-tls\") pod \"metrics-server-5b8b89bf95-slsg6\" (UID: \"2a7ca0b4-ec03-407e-bade-0a533e7c1b15\") " pod="openshift-monitoring/metrics-server-5b8b89bf95-slsg6"
Sep 16 17:27:37.308614 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:37.308587    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vw8nj\" (UniqueName: \"kubernetes.io/projected/2a7ca0b4-ec03-407e-bade-0a533e7c1b15-kube-api-access-vw8nj\") pod \"metrics-server-5b8b89bf95-slsg6\" (UID: \"2a7ca0b4-ec03-407e-bade-0a533e7c1b15\") " pod="openshift-monitoring/metrics-server-5b8b89bf95-slsg6"
Sep 16 17:27:37.338690 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:37.338651    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-d2fsj"]
Sep 16 17:27:37.343126 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:37.343101    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-d2fsj"
Sep 16 17:27:37.345754 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:37.345730    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-75rnr\""
Sep 16 17:27:37.345878 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:37.345789    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Sep 16 17:27:37.352440 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:37.352411    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-d2fsj"]
Sep 16 17:27:37.394985 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:37.394906    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwh4t\" (UniqueName: \"kubernetes.io/projected/5a36f055-8d5d-4bda-9a14-019707863d03-kube-api-access-pwh4t\") pod \"monitoring-plugin-5d85778b9d-d2fsj\" (UID: \"5a36f055-8d5d-4bda-9a14-019707863d03\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-d2fsj"
Sep 16 17:27:37.395147 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:37.395025    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5a36f055-8d5d-4bda-9a14-019707863d03-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-d2fsj\" (UID: \"5a36f055-8d5d-4bda-9a14-019707863d03\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-d2fsj"
Sep 16 17:27:37.445628 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:37.445582    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-5b8b89bf95-slsg6"
Sep 16 17:27:37.495980 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:37.495948    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5a36f055-8d5d-4bda-9a14-019707863d03-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-d2fsj\" (UID: \"5a36f055-8d5d-4bda-9a14-019707863d03\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-d2fsj"
Sep 16 17:27:37.496144 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:37.496045    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwh4t\" (UniqueName: \"kubernetes.io/projected/5a36f055-8d5d-4bda-9a14-019707863d03-kube-api-access-pwh4t\") pod \"monitoring-plugin-5d85778b9d-d2fsj\" (UID: \"5a36f055-8d5d-4bda-9a14-019707863d03\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-d2fsj"
Sep 16 17:27:37.496144 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:27:37.496106    2579 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found
Sep 16 17:27:37.496228 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:27:37.496175    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5a36f055-8d5d-4bda-9a14-019707863d03-monitoring-plugin-cert podName:5a36f055-8d5d-4bda-9a14-019707863d03 nodeName:}" failed. No retries permitted until 2026-09-16 17:27:37.996158985 +0000 UTC m=+170.466157118 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/5a36f055-8d5d-4bda-9a14-019707863d03-monitoring-plugin-cert") pod "monitoring-plugin-5d85778b9d-d2fsj" (UID: "5a36f055-8d5d-4bda-9a14-019707863d03") : secret "monitoring-plugin-cert" not found
Sep 16 17:27:37.506823 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:37.506791    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwh4t\" (UniqueName: \"kubernetes.io/projected/5a36f055-8d5d-4bda-9a14-019707863d03-kube-api-access-pwh4t\") pod \"monitoring-plugin-5d85778b9d-d2fsj\" (UID: \"5a36f055-8d5d-4bda-9a14-019707863d03\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-d2fsj"
Sep 16 17:27:37.534044 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:37.534014    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-9d65f485-4kk8t"]
Sep 16 17:27:37.566895 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:37.566821    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-97c9d775b-jcdpd"]
Sep 16 17:27:37.571781 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:37.571658    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-97c9d775b-jcdpd"
Sep 16 17:27:37.583496 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:37.583471    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-97c9d775b-jcdpd"]
Sep 16 17:27:37.697365 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:37.697338    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/817c0617-8912-4e88-850e-f8b89ac8b191-service-ca\") pod \"console-97c9d775b-jcdpd\" (UID: \"817c0617-8912-4e88-850e-f8b89ac8b191\") " pod="openshift-console/console-97c9d775b-jcdpd"
Sep 16 17:27:37.697470 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:37.697385    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/817c0617-8912-4e88-850e-f8b89ac8b191-console-oauth-config\") pod \"console-97c9d775b-jcdpd\" (UID: \"817c0617-8912-4e88-850e-f8b89ac8b191\") " pod="openshift-console/console-97c9d775b-jcdpd"
Sep 16 17:27:37.697470 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:37.697428    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/817c0617-8912-4e88-850e-f8b89ac8b191-console-serving-cert\") pod \"console-97c9d775b-jcdpd\" (UID: \"817c0617-8912-4e88-850e-f8b89ac8b191\") " pod="openshift-console/console-97c9d775b-jcdpd"
Sep 16 17:27:37.697602 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:37.697487    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/817c0617-8912-4e88-850e-f8b89ac8b191-console-config\") pod \"console-97c9d775b-jcdpd\" (UID: \"817c0617-8912-4e88-850e-f8b89ac8b191\") " pod="openshift-console/console-97c9d775b-jcdpd"
Sep 16 17:27:37.697602 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:37.697536    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/817c0617-8912-4e88-850e-f8b89ac8b191-oauth-serving-cert\") pod \"console-97c9d775b-jcdpd\" (UID: \"817c0617-8912-4e88-850e-f8b89ac8b191\") " pod="openshift-console/console-97c9d775b-jcdpd"
Sep 16 17:27:37.697602 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:37.697577    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwt4v\" (UniqueName: \"kubernetes.io/projected/817c0617-8912-4e88-850e-f8b89ac8b191-kube-api-access-cwt4v\") pod \"console-97c9d775b-jcdpd\" (UID: \"817c0617-8912-4e88-850e-f8b89ac8b191\") " pod="openshift-console/console-97c9d775b-jcdpd"
Sep 16 17:27:37.697751 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:37.697622    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/817c0617-8912-4e88-850e-f8b89ac8b191-trusted-ca-bundle\") pod \"console-97c9d775b-jcdpd\" (UID: \"817c0617-8912-4e88-850e-f8b89ac8b191\") " pod="openshift-console/console-97c9d775b-jcdpd"
Sep 16 17:27:37.699673 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:37.699599    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-fbrpx" event={"ID":"f8ebccfe-c8a8-4a90-8cad-c764e3ee9956","Type":"ContainerStarted","Data":"1d43eaf5fcc4cae763f04b3fe416fd91eb6ce6e229e193ef897ddde4b30fdeff"}
Sep 16 17:27:37.798834 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:37.798799    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/817c0617-8912-4e88-850e-f8b89ac8b191-oauth-serving-cert\") pod \"console-97c9d775b-jcdpd\" (UID: \"817c0617-8912-4e88-850e-f8b89ac8b191\") " pod="openshift-console/console-97c9d775b-jcdpd"
Sep 16 17:27:37.798834 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:37.798875    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwt4v\" (UniqueName: \"kubernetes.io/projected/817c0617-8912-4e88-850e-f8b89ac8b191-kube-api-access-cwt4v\") pod \"console-97c9d775b-jcdpd\" (UID: \"817c0617-8912-4e88-850e-f8b89ac8b191\") " pod="openshift-console/console-97c9d775b-jcdpd"
Sep 16 17:27:37.799159 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:37.798920    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/817c0617-8912-4e88-850e-f8b89ac8b191-trusted-ca-bundle\") pod \"console-97c9d775b-jcdpd\" (UID: \"817c0617-8912-4e88-850e-f8b89ac8b191\") " pod="openshift-console/console-97c9d775b-jcdpd"
Sep 16 17:27:37.799159 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:37.798986    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/817c0617-8912-4e88-850e-f8b89ac8b191-service-ca\") pod \"console-97c9d775b-jcdpd\" (UID: \"817c0617-8912-4e88-850e-f8b89ac8b191\") " pod="openshift-console/console-97c9d775b-jcdpd"
Sep 16 17:27:37.799159 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:37.799010    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/817c0617-8912-4e88-850e-f8b89ac8b191-console-oauth-config\") pod \"console-97c9d775b-jcdpd\" (UID: \"817c0617-8912-4e88-850e-f8b89ac8b191\") " pod="openshift-console/console-97c9d775b-jcdpd"
Sep 16 17:27:37.799159 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:37.799048    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/817c0617-8912-4e88-850e-f8b89ac8b191-console-serving-cert\") pod \"console-97c9d775b-jcdpd\" (UID: \"817c0617-8912-4e88-850e-f8b89ac8b191\") " pod="openshift-console/console-97c9d775b-jcdpd"
Sep 16 17:27:37.799159 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:37.799094    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/817c0617-8912-4e88-850e-f8b89ac8b191-console-config\") pod \"console-97c9d775b-jcdpd\" (UID: \"817c0617-8912-4e88-850e-f8b89ac8b191\") " pod="openshift-console/console-97c9d775b-jcdpd"
Sep 16 17:27:37.799792 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:37.799737    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/817c0617-8912-4e88-850e-f8b89ac8b191-service-ca\") pod \"console-97c9d775b-jcdpd\" (UID: \"817c0617-8912-4e88-850e-f8b89ac8b191\") " pod="openshift-console/console-97c9d775b-jcdpd"
Sep 16 17:27:37.799945 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:37.799819    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/817c0617-8912-4e88-850e-f8b89ac8b191-console-config\") pod \"console-97c9d775b-jcdpd\" (UID: \"817c0617-8912-4e88-850e-f8b89ac8b191\") " pod="openshift-console/console-97c9d775b-jcdpd"
Sep 16 17:27:37.799945 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:37.799821    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/817c0617-8912-4e88-850e-f8b89ac8b191-oauth-serving-cert\") pod \"console-97c9d775b-jcdpd\" (UID: \"817c0617-8912-4e88-850e-f8b89ac8b191\") " pod="openshift-console/console-97c9d775b-jcdpd"
Sep 16 17:27:37.800066 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:37.799941    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/817c0617-8912-4e88-850e-f8b89ac8b191-trusted-ca-bundle\") pod \"console-97c9d775b-jcdpd\" (UID: \"817c0617-8912-4e88-850e-f8b89ac8b191\") " pod="openshift-console/console-97c9d775b-jcdpd"
Sep 16 17:27:37.801769 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:37.801740    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/817c0617-8912-4e88-850e-f8b89ac8b191-console-oauth-config\") pod \"console-97c9d775b-jcdpd\" (UID: \"817c0617-8912-4e88-850e-f8b89ac8b191\") " pod="openshift-console/console-97c9d775b-jcdpd"
Sep 16 17:27:37.801900 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:37.801808    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/817c0617-8912-4e88-850e-f8b89ac8b191-console-serving-cert\") pod \"console-97c9d775b-jcdpd\" (UID: \"817c0617-8912-4e88-850e-f8b89ac8b191\") " pod="openshift-console/console-97c9d775b-jcdpd"
Sep 16 17:27:37.810140 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:37.810114    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwt4v\" (UniqueName: \"kubernetes.io/projected/817c0617-8912-4e88-850e-f8b89ac8b191-kube-api-access-cwt4v\") pod \"console-97c9d775b-jcdpd\" (UID: \"817c0617-8912-4e88-850e-f8b89ac8b191\") " pod="openshift-console/console-97c9d775b-jcdpd"
Sep 16 17:27:37.891102 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:37.891058    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-97c9d775b-jcdpd"
Sep 16 17:27:37.893057 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:27:37.893017    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27605ac2_4ce2_4847_b391_7f803ed3f809.slice/crio-fca9a82663447e8a4234c6bb2bf95d3b93950d508d9f5132e40bf18c2ea82695 WatchSource:0}: Error finding container fca9a82663447e8a4234c6bb2bf95d3b93950d508d9f5132e40bf18c2ea82695: Status 404 returned error can't find the container with id fca9a82663447e8a4234c6bb2bf95d3b93950d508d9f5132e40bf18c2ea82695
Sep 16 17:27:37.894150 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:37.894116    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-654695c99f-4jmw9"]
Sep 16 17:27:37.901909 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:37.901867    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-5b8b89bf95-slsg6"]
Sep 16 17:27:37.908259 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:37.908229    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-d77b4b4b-2f2bv"]
Sep 16 17:27:37.911212 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:37.911189    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-9cbqd"]
Sep 16 17:27:37.911448 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:27:37.911416    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda639ab67_695c_4ec2_af20_1882ac49c4df.slice/crio-b005537ba19bb3fc26845aa1178be636a2de791e7517a9a918788a44e522fe10 WatchSource:0}: Error finding container b005537ba19bb3fc26845aa1178be636a2de791e7517a9a918788a44e522fe10: Status 404 returned error can't find the container with id b005537ba19bb3fc26845aa1178be636a2de791e7517a9a918788a44e522fe10
Sep 16 17:27:37.913108 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:27:37.912922    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c32493a_a573_4c14_9e85_9f0faef07cd5.slice/crio-da4421835201ef23e8eab2beb4acd37485f0e1271cfe243224cdc849fe889b60 WatchSource:0}: Error finding container da4421835201ef23e8eab2beb4acd37485f0e1271cfe243224cdc849fe889b60: Status 404 returned error can't find the container with id da4421835201ef23e8eab2beb4acd37485f0e1271cfe243224cdc849fe889b60
Sep 16 17:27:37.915287 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:27:37.915242    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a7ca0b4_ec03_407e_bade_0a533e7c1b15.slice/crio-a7f20cd9071e49c5d8a1e4d5c9a9d178f0c945bfabb4c16b30a4521ac81188de WatchSource:0}: Error finding container a7f20cd9071e49c5d8a1e4d5c9a9d178f0c945bfabb4c16b30a4521ac81188de: Status 404 returned error can't find the container with id a7f20cd9071e49c5d8a1e4d5c9a9d178f0c945bfabb4c16b30a4521ac81188de
Sep 16 17:27:37.917832 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:37.917672    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 17:27:38.000984 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:38.000937    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5a36f055-8d5d-4bda-9a14-019707863d03-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-d2fsj\" (UID: \"5a36f055-8d5d-4bda-9a14-019707863d03\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-d2fsj"
Sep 16 17:27:38.008230 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:38.008010    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5a36f055-8d5d-4bda-9a14-019707863d03-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-d2fsj\" (UID: \"5a36f055-8d5d-4bda-9a14-019707863d03\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-d2fsj"
Sep 16 17:27:38.096896 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:38.096051    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xfgjq"
Sep 16 17:27:38.204110 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:38.204010    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-97c9d775b-jcdpd"]
Sep 16 17:27:38.209932 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:27:38.209898    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod817c0617_8912_4e88_850e_f8b89ac8b191.slice/crio-46f7accf1e49ad536e3b00b48aeb5018d47f1a45abcd9f5f8ff5ffc2edc68650 WatchSource:0}: Error finding container 46f7accf1e49ad536e3b00b48aeb5018d47f1a45abcd9f5f8ff5ffc2edc68650: Status 404 returned error can't find the container with id 46f7accf1e49ad536e3b00b48aeb5018d47f1a45abcd9f5f8ff5ffc2edc68650
Sep 16 17:27:38.255467 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:38.255380    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-d2fsj"
Sep 16 17:27:38.569751 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:38.569712    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-d2fsj"]
Sep 16 17:27:38.611812 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:27:38.611765    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a36f055_8d5d_4bda_9a14_019707863d03.slice/crio-3ba5062a36f57b5b9c213890e28e10df99794ec37ff9e3f4e3b43ebf54943bc2 WatchSource:0}: Error finding container 3ba5062a36f57b5b9c213890e28e10df99794ec37ff9e3f4e3b43ebf54943bc2: Status 404 returned error can't find the container with id 3ba5062a36f57b5b9c213890e28e10df99794ec37ff9e3f4e3b43ebf54943bc2
Sep 16 17:27:38.716568 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:38.715796    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-97c9d775b-jcdpd" event={"ID":"817c0617-8912-4e88-850e-f8b89ac8b191","Type":"ContainerStarted","Data":"a5041b5023c85ddcf8e562e0dd26226b460a49f00e2003f9ae4d0a3a650f0c6a"}
Sep 16 17:27:38.716568 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:38.715878    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-97c9d775b-jcdpd" event={"ID":"817c0617-8912-4e88-850e-f8b89ac8b191","Type":"ContainerStarted","Data":"46f7accf1e49ad536e3b00b48aeb5018d47f1a45abcd9f5f8ff5ffc2edc68650"}
Sep 16 17:27:38.718153 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:38.718055    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-9cbqd" event={"ID":"a639ab67-695c-4ec2-af20-1882ac49c4df","Type":"ContainerStarted","Data":"b005537ba19bb3fc26845aa1178be636a2de791e7517a9a918788a44e522fe10"}
Sep 16 17:27:38.720379 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:38.720349    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-76f99584f4-xf777" event={"ID":"ff8acb9f-034a-4aab-aed4-aebb3df42f07","Type":"ContainerStarted","Data":"3cf5e23c341c2d703c61ed53f58134c00c527e64eb0f028e094a5f359e6a8567"}
Sep 16 17:27:38.721115 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:38.721051    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-76f99584f4-xf777"
Sep 16 17:27:38.724060 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:38.724013    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-5b8b89bf95-slsg6" event={"ID":"2a7ca0b4-ec03-407e-bade-0a533e7c1b15","Type":"ContainerStarted","Data":"a7f20cd9071e49c5d8a1e4d5c9a9d178f0c945bfabb4c16b30a4521ac81188de"}
Sep 16 17:27:38.725428 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:38.725406    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-d2fsj" event={"ID":"5a36f055-8d5d-4bda-9a14-019707863d03","Type":"ContainerStarted","Data":"3ba5062a36f57b5b9c213890e28e10df99794ec37ff9e3f4e3b43ebf54943bc2"}
Sep 16 17:27:38.728065 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:38.727725    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a2ad2862-5d15-46b2-aaeb-a96ad6af3255","Type":"ContainerStarted","Data":"eeb5ab817b93d97099c1c9ab181acf794ddc2ffa99d2b22d5edccb2c1fbc3d17"}
Sep 16 17:27:38.730410 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:38.730383    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-654695c99f-4jmw9" event={"ID":"27605ac2-4ce2-4847-b391-7f803ed3f809","Type":"ContainerStarted","Data":"fca9a82663447e8a4234c6bb2bf95d3b93950d508d9f5132e40bf18c2ea82695"}
Sep 16 17:27:38.733306 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:38.733278    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-9d65f485-4kk8t" event={"ID":"9ba8e716-8f2b-4fe4-a140-707f7335fbef","Type":"ContainerStarted","Data":"f4ab3b3e034a359843a43c3743093ace442a2633b16296f6a1fabc2e1a31c03c"}
Sep 16 17:27:38.736088 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:38.736056    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-d77b4b4b-2f2bv" event={"ID":"3c32493a-a573-4c14-9e85-9f0faef07cd5","Type":"ContainerStarted","Data":"e02951a32900a7648555dbe0a485791c10f0ae2d2320aa3c9a0b424ecba12261"}
Sep 16 17:27:38.736263 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:38.736116    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-d77b4b4b-2f2bv" event={"ID":"3c32493a-a573-4c14-9e85-9f0faef07cd5","Type":"ContainerStarted","Data":"da4421835201ef23e8eab2beb4acd37485f0e1271cfe243224cdc849fe889b60"}
Sep 16 17:27:38.740093 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:38.740054    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-76f99584f4-xf777"
Sep 16 17:27:38.799615 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:38.799547    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-9d65f485-4kk8t" podStartSLOduration=2.861593006 podStartE2EDuration="11.799522304s" totalImagesPullingTime="8.937929298s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:27:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:27:38.798386825 +0000 UTC m=+171.268384963" watchObservedRunningTime="2026-09-16 17:27:38.799522304 +0000 UTC m=+171.269520442"
Sep 16 17:27:38.800765 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:38.800726    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-97c9d775b-jcdpd" podStartSLOduration=1.800710664 podStartE2EDuration="1.800710664s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:27:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:27:38.756238973 +0000 UTC m=+171.226237108" watchObservedRunningTime="2026-09-16 17:27:38.800710664 +0000 UTC m=+171.270708793"
Sep 16 17:27:38.883056 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:38.882989    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-d77b4b4b-2f2bv" podStartSLOduration=4.882965885 podStartE2EDuration="4.882965885s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:27:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:27:38.848240791 +0000 UTC m=+171.318238929" watchObservedRunningTime="2026-09-16 17:27:38.882965885 +0000 UTC m=+171.352964021"
Sep 16 17:27:39.749167 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:39.749128    2579 generic.go:356] "Generic (PLEG): container finished" podID="f8ebccfe-c8a8-4a90-8cad-c764e3ee9956" containerID="864468680e91bf932936bc3768eeb3f31ac153fb9eeba571765703983e22ca37" exitCode=0
Sep 16 17:27:39.750404 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:39.750373    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-fbrpx" event={"ID":"f8ebccfe-c8a8-4a90-8cad-c764e3ee9956","Type":"ContainerDied","Data":"864468680e91bf932936bc3768eeb3f31ac153fb9eeba571765703983e22ca37"}
Sep 16 17:27:39.807163 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:39.805260    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-76f99584f4-xf777" podStartSLOduration=3.467030651 podStartE2EDuration="22.805235469s" totalImagesPullingTime="19.338204818s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:27:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:27:38.883596451 +0000 UTC m=+171.353594587" watchObservedRunningTime="2026-09-16 17:27:39.805235469 +0000 UTC m=+172.275233608"
Sep 16 17:27:40.756972 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:40.756888    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-fbrpx" event={"ID":"f8ebccfe-c8a8-4a90-8cad-c764e3ee9956","Type":"ContainerStarted","Data":"7cd52247ec9f07d13eec759a7bcd0f9fda0c5364ca88ea55174d570232e68e2f"}
Sep 16 17:27:44.649935 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:44.649832    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-d77b4b4b-2f2bv"
Sep 16 17:27:44.650423 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:44.649961    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-d77b4b4b-2f2bv"
Sep 16 17:27:44.655914 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:44.655888    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-d77b4b4b-2f2bv"
Sep 16 17:27:44.777627 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:44.777592    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-fbrpx" event={"ID":"f8ebccfe-c8a8-4a90-8cad-c764e3ee9956","Type":"ContainerStarted","Data":"6e0710fff96f84352962bd2b27617d23c64d71e86239b32b6ea3ea23f0fd4f1e"}
Sep 16 17:27:44.780044 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:44.780012    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-9cbqd" event={"ID":"a639ab67-695c-4ec2-af20-1882ac49c4df","Type":"ContainerStarted","Data":"fb7936f37c53531b2ed130425ac108a7f6e68dc431cc953853570d5afa44c01d"}
Sep 16 17:27:44.780192 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:44.780069    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-9cbqd" event={"ID":"a639ab67-695c-4ec2-af20-1882ac49c4df","Type":"ContainerStarted","Data":"c646c9c6037e1c7c5b596615e4534cc91865af1c9db62e123d421f915e10f40b"}
Sep 16 17:27:44.780192 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:44.780096    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-9cbqd"
Sep 16 17:27:44.781874 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:44.781825    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-5b8b89bf95-slsg6" event={"ID":"2a7ca0b4-ec03-407e-bade-0a533e7c1b15","Type":"ContainerStarted","Data":"40acffdd29ab005a78846aba26eca0ee4ec08fa2a9ce373789b3b3fd1bd4f7f8"}
Sep 16 17:27:44.783598 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:44.783573    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-d2fsj" event={"ID":"5a36f055-8d5d-4bda-9a14-019707863d03","Type":"ContainerStarted","Data":"a4c7d968f18c91000b5701720a4b712588d7ac74f9a78b3b4f92b703f4a7ccd6"}
Sep 16 17:27:44.783710 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:44.783625    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-d2fsj"
Sep 16 17:27:44.785446 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:44.785419    2579 generic.go:356] "Generic (PLEG): container finished" podID="a2ad2862-5d15-46b2-aaeb-a96ad6af3255" containerID="8bc46bc5b55389fcb60ab6a68987c6fdc2ca9ce5cb68aad0b86df70da96c7da1" exitCode=0
Sep 16 17:27:44.785567 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:44.785536    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a2ad2862-5d15-46b2-aaeb-a96ad6af3255","Type":"ContainerDied","Data":"8bc46bc5b55389fcb60ab6a68987c6fdc2ca9ce5cb68aad0b86df70da96c7da1"}
Sep 16 17:27:44.789248 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:44.789224    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-654695c99f-4jmw9" event={"ID":"27605ac2-4ce2-4847-b391-7f803ed3f809","Type":"ContainerStarted","Data":"31d6638005ed0fe4671c1f42683fbdeaf4bedb2dff9826293b09729d90e9905d"}
Sep 16 17:27:44.789337 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:44.789303    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-654695c99f-4jmw9" event={"ID":"27605ac2-4ce2-4847-b391-7f803ed3f809","Type":"ContainerStarted","Data":"a6359de8101bb62ba94e3b80ffacbc57231b32a70c6b663331738a3ec750b0c3"}
Sep 16 17:27:44.789394 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:44.789349    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-654695c99f-4jmw9" event={"ID":"27605ac2-4ce2-4847-b391-7f803ed3f809","Type":"ContainerStarted","Data":"f3c46ab9872ad35163f83c00d7a85efc4eb8ddeb7d11177ca7d598adb3b7b46c"}
Sep 16 17:27:44.790605 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:44.790585    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-d2fsj"
Sep 16 17:27:44.794061 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:44.794015    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-d77b4b4b-2f2bv"
Sep 16 17:27:44.814208 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:44.814152    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-fbrpx" podStartSLOduration=11.646392862999999 podStartE2EDuration="12.814130827s" totalImagesPullingTime="1.073714246s" totalInitContainerRuntime="94.023718ms" isStatefulPod=true podCreationTimestamp="2026-09-16 17:27:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:27:44.813308194 +0000 UTC m=+177.283306443" watchObservedRunningTime="2026-09-16 17:27:44.814130827 +0000 UTC m=+177.284128964"
Sep 16 17:27:44.894869 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:44.894792    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-9cbqd" podStartSLOduration=136.987613587 podStartE2EDuration="2m22.894773898s" totalImagesPullingTime="5.907160311s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:25:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:27:44.893743001 +0000 UTC m=+177.363741138" watchObservedRunningTime="2026-09-16 17:27:44.894773898 +0000 UTC m=+177.364772078"
Sep 16 17:27:44.998765 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:44.998641    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-5b8b89bf95-slsg6" podStartSLOduration=2.097291189 podStartE2EDuration="7.998615872s" totalImagesPullingTime="5.901324683s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:27:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:27:44.989077948 +0000 UTC m=+177.459076085" watchObservedRunningTime="2026-09-16 17:27:44.998615872 +0000 UTC m=+177.468614009"
Sep 16 17:27:45.030652 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:45.030595    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-d2fsj" podStartSLOduration=2.822991608 podStartE2EDuration="8.030571991s" totalImagesPullingTime="5.207580383s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:27:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:27:45.02955833 +0000 UTC m=+177.499556466" watchObservedRunningTime="2026-09-16 17:27:45.030571991 +0000 UTC m=+177.500570126"
Sep 16 17:27:46.802330 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:46.802292    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-654695c99f-4jmw9" event={"ID":"27605ac2-4ce2-4847-b391-7f803ed3f809","Type":"ContainerStarted","Data":"d1f7c313da3d962cfd1761eea163bd55d2b38c067c50dbcf980c656a9c1c8afe"}
Sep 16 17:27:46.802799 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:46.802365    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-654695c99f-4jmw9" event={"ID":"27605ac2-4ce2-4847-b391-7f803ed3f809","Type":"ContainerStarted","Data":"691ecf694a61fa0d2265eeb5de5b51a770d0a5a0726245584d3e3e9a683af4a6"}
Sep 16 17:27:46.802799 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:46.802393    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-654695c99f-4jmw9" event={"ID":"27605ac2-4ce2-4847-b391-7f803ed3f809","Type":"ContainerStarted","Data":"9c6028c22df2a434e39d809c8ceb533b8afa372f39cd1a996708134a3dc41e95"}
Sep 16 17:27:47.808895 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:47.808868    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a2ad2862-5d15-46b2-aaeb-a96ad6af3255","Type":"ContainerStarted","Data":"915c986887654ba3ce50062e4938339cb3504d0da594ebd5ad60868f60b7c526"}
Sep 16 17:27:47.809214 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:47.808961    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a2ad2862-5d15-46b2-aaeb-a96ad6af3255","Type":"ContainerStarted","Data":"84faa18e8faf36d1f969d8b0e7cd3ab717e604a6c2ae21561e670c53dfe1613f"}
Sep 16 17:27:47.809214 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:47.808988    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a2ad2862-5d15-46b2-aaeb-a96ad6af3255","Type":"ContainerStarted","Data":"db923488d4f7ebd57a238df7f99586eefd464ab5be7c5284676df6b897c3b734"}
Sep 16 17:27:47.809214 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:47.809017    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a2ad2862-5d15-46b2-aaeb-a96ad6af3255","Type":"ContainerStarted","Data":"7154276c8a3fdc2fc102e0015c16c36591e850d44a73305fcf9d4514c616d25f"}
Sep 16 17:27:47.809214 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:47.809039    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a2ad2862-5d15-46b2-aaeb-a96ad6af3255","Type":"ContainerStarted","Data":"6433d2236aad85b04301e28bdad7e4af1109f4cb9ac844bc47af43a367573ba9"}
Sep 16 17:27:47.809214 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:47.809198    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-654695c99f-4jmw9"
Sep 16 17:27:47.891629 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:47.891534    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-97c9d775b-jcdpd"
Sep 16 17:27:47.891629 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:47.891606    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-97c9d775b-jcdpd"
Sep 16 17:27:47.896481 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:47.896456    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-97c9d775b-jcdpd"
Sep 16 17:27:47.923400 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:47.922739    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-654695c99f-4jmw9" podStartSLOduration=5.580876953 podStartE2EDuration="12.922716058s" totalImagesPullingTime="7.341839105s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:27:35 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:27:46.833337468 +0000 UTC m=+179.303335603" watchObservedRunningTime="2026-09-16 17:27:47.922716058 +0000 UTC m=+180.392714194"
Sep 16 17:27:48.331281 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:48.331241    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-9d65f485-4kk8t"
Sep 16 17:27:48.816562 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:48.816523    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a2ad2862-5d15-46b2-aaeb-a96ad6af3255","Type":"ContainerStarted","Data":"d68bec048ec2edc292dd1029ceba90f9feb844152cf756ad0de79d86c8636422"}
Sep 16 17:27:48.821308 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:48.821279    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-97c9d775b-jcdpd"
Sep 16 17:27:48.875707 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:48.875649    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=7.64248665 podStartE2EDuration="15.875630418s" totalImagesPullingTime="8.19776108s" totalInitContainerRuntime="35.382688ms" isStatefulPod=false podCreationTimestamp="2026-09-16 17:27:33 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:27:48.872981868 +0000 UTC m=+181.342980003" watchObservedRunningTime="2026-09-16 17:27:48.875630418 +0000 UTC m=+181.345628553"
Sep 16 17:27:48.957368 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:48.957335    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-d77b4b4b-2f2bv"]
Sep 16 17:27:53.824135 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:53.824107    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-654695c99f-4jmw9"
Sep 16 17:27:54.798660 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:54.798632    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-9cbqd"
Sep 16 17:27:57.445704 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:57.445664    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-5b8b89bf95-slsg6"
Sep 16 17:27:57.446145 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:57.445730    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-5b8b89bf95-slsg6"
Sep 16 17:27:57.854749 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:57.854718    2579 generic.go:356] "Generic (PLEG): container finished" podID="51b86650-5910-4bc0-9832-dc967737cbea" containerID="8243e8741a4c7e81c281709658a5bab4b246c456435fa1cd3eebd430a8f28afc" exitCode=0
Sep 16 17:27:57.854749 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:57.854759    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-f7kcw" event={"ID":"51b86650-5910-4bc0-9832-dc967737cbea","Type":"ContainerDied","Data":"8243e8741a4c7e81c281709658a5bab4b246c456435fa1cd3eebd430a8f28afc"}
Sep 16 17:27:57.855176 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:57.855159    2579 scope.go:117] "RemoveContainer" containerID="8243e8741a4c7e81c281709658a5bab4b246c456435fa1cd3eebd430a8f28afc"
Sep 16 17:27:58.860083 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:27:58.860049    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-f7kcw" event={"ID":"51b86650-5910-4bc0-9832-dc967737cbea","Type":"ContainerStarted","Data":"772b7bf35c8269140b4450041f9b9c6716d38fd6025af61a313859a136952884"}
Sep 16 17:28:03.763357 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:03.763267    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-9d65f485-4kk8t" podUID="9ba8e716-8f2b-4fe4-a140-707f7335fbef" containerName="console" containerID="cri-o://f4ab3b3e034a359843a43c3743093ace442a2633b16296f6a1fabc2e1a31c03c" gracePeriod=15
Sep 16 17:28:03.880312 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:03.880285    2579 generic.go:356] "Generic (PLEG): container finished" podID="9ba8e716-8f2b-4fe4-a140-707f7335fbef" containerID="f4ab3b3e034a359843a43c3743093ace442a2633b16296f6a1fabc2e1a31c03c" exitCode=0
Sep 16 17:28:03.880406 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:03.880365    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-9d65f485-4kk8t" event={"ID":"9ba8e716-8f2b-4fe4-a140-707f7335fbef","Type":"ContainerDied","Data":"f4ab3b3e034a359843a43c3743093ace442a2633b16296f6a1fabc2e1a31c03c"}
Sep 16 17:28:03.951109 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:03.951077    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-9cbqd_a639ab67-695c-4ec2-af20-1882ac49c4df/dns/0.log"
Sep 16 17:28:04.006540 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:04.006516    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-9d65f485-4kk8t"
Sep 16 17:28:04.063585 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:04.063553    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9ba8e716-8f2b-4fe4-a140-707f7335fbef-console-config\") pod \"9ba8e716-8f2b-4fe4-a140-707f7335fbef\" (UID: \"9ba8e716-8f2b-4fe4-a140-707f7335fbef\") "
Sep 16 17:28:04.063767 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:04.063607    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9ba8e716-8f2b-4fe4-a140-707f7335fbef-service-ca\") pod \"9ba8e716-8f2b-4fe4-a140-707f7335fbef\" (UID: \"9ba8e716-8f2b-4fe4-a140-707f7335fbef\") "
Sep 16 17:28:04.063767 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:04.063626    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9ba8e716-8f2b-4fe4-a140-707f7335fbef-oauth-serving-cert\") pod \"9ba8e716-8f2b-4fe4-a140-707f7335fbef\" (UID: \"9ba8e716-8f2b-4fe4-a140-707f7335fbef\") "
Sep 16 17:28:04.063767 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:04.063745    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9ba8e716-8f2b-4fe4-a140-707f7335fbef-console-serving-cert\") pod \"9ba8e716-8f2b-4fe4-a140-707f7335fbef\" (UID: \"9ba8e716-8f2b-4fe4-a140-707f7335fbef\") "
Sep 16 17:28:04.063962 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:04.063816    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9ba8e716-8f2b-4fe4-a140-707f7335fbef-console-oauth-config\") pod \"9ba8e716-8f2b-4fe4-a140-707f7335fbef\" (UID: \"9ba8e716-8f2b-4fe4-a140-707f7335fbef\") "
Sep 16 17:28:04.063962 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:04.063886    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zcl4r\" (UniqueName: \"kubernetes.io/projected/9ba8e716-8f2b-4fe4-a140-707f7335fbef-kube-api-access-zcl4r\") pod \"9ba8e716-8f2b-4fe4-a140-707f7335fbef\" (UID: \"9ba8e716-8f2b-4fe4-a140-707f7335fbef\") "
Sep 16 17:28:04.064055 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:04.064026    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ba8e716-8f2b-4fe4-a140-707f7335fbef-console-config" (OuterVolumeSpecName: "console-config") pod "9ba8e716-8f2b-4fe4-a140-707f7335fbef" (UID: "9ba8e716-8f2b-4fe4-a140-707f7335fbef"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 17:28:04.064055 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:04.064034    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ba8e716-8f2b-4fe4-a140-707f7335fbef-service-ca" (OuterVolumeSpecName: "service-ca") pod "9ba8e716-8f2b-4fe4-a140-707f7335fbef" (UID: "9ba8e716-8f2b-4fe4-a140-707f7335fbef"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 17:28:04.064150 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:04.064048    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ba8e716-8f2b-4fe4-a140-707f7335fbef-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "9ba8e716-8f2b-4fe4-a140-707f7335fbef" (UID: "9ba8e716-8f2b-4fe4-a140-707f7335fbef"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 17:28:04.064207 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:04.064189    2579 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9ba8e716-8f2b-4fe4-a140-707f7335fbef-console-config\") on node \"ip-10-0-138-73.ec2.internal\" DevicePath \"\""
Sep 16 17:28:04.064207 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:04.064206    2579 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9ba8e716-8f2b-4fe4-a140-707f7335fbef-service-ca\") on node \"ip-10-0-138-73.ec2.internal\" DevicePath \"\""
Sep 16 17:28:04.064323 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:04.064215    2579 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9ba8e716-8f2b-4fe4-a140-707f7335fbef-oauth-serving-cert\") on node \"ip-10-0-138-73.ec2.internal\" DevicePath \"\""
Sep 16 17:28:04.066068 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:04.066038    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ba8e716-8f2b-4fe4-a140-707f7335fbef-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "9ba8e716-8f2b-4fe4-a140-707f7335fbef" (UID: "9ba8e716-8f2b-4fe4-a140-707f7335fbef"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 17:28:04.066068 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:04.066052    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ba8e716-8f2b-4fe4-a140-707f7335fbef-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "9ba8e716-8f2b-4fe4-a140-707f7335fbef" (UID: "9ba8e716-8f2b-4fe4-a140-707f7335fbef"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 17:28:04.066068 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:04.066047    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ba8e716-8f2b-4fe4-a140-707f7335fbef-kube-api-access-zcl4r" (OuterVolumeSpecName: "kube-api-access-zcl4r") pod "9ba8e716-8f2b-4fe4-a140-707f7335fbef" (UID: "9ba8e716-8f2b-4fe4-a140-707f7335fbef"). InnerVolumeSpecName "kube-api-access-zcl4r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 17:28:04.149470 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:04.149441    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-9cbqd_a639ab67-695c-4ec2-af20-1882ac49c4df/kube-rbac-proxy/0.log"
Sep 16 17:28:04.164713 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:04.164681    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zcl4r\" (UniqueName: \"kubernetes.io/projected/9ba8e716-8f2b-4fe4-a140-707f7335fbef-kube-api-access-zcl4r\") on node \"ip-10-0-138-73.ec2.internal\" DevicePath \"\""
Sep 16 17:28:04.164713 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:04.164710    2579 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9ba8e716-8f2b-4fe4-a140-707f7335fbef-console-serving-cert\") on node \"ip-10-0-138-73.ec2.internal\" DevicePath \"\""
Sep 16 17:28:04.164876 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:04.164719    2579 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9ba8e716-8f2b-4fe4-a140-707f7335fbef-console-oauth-config\") on node \"ip-10-0-138-73.ec2.internal\" DevicePath \"\""
Sep 16 17:28:04.885240 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:04.885196    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-9d65f485-4kk8t" event={"ID":"9ba8e716-8f2b-4fe4-a140-707f7335fbef","Type":"ContainerDied","Data":"8d7b0c907df37d1911b750ef633997f43591ed03408ec15e336693a6bead0dc6"}
Sep 16 17:28:04.885686 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:04.885259    2579 scope.go:117] "RemoveContainer" containerID="f4ab3b3e034a359843a43c3743093ace442a2633b16296f6a1fabc2e1a31c03c"
Sep 16 17:28:04.885686 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:04.885212    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-9d65f485-4kk8t"
Sep 16 17:28:04.903929 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:04.903899    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-9d65f485-4kk8t"]
Sep 16 17:28:04.907261 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:04.907240    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-9d65f485-4kk8t"]
Sep 16 17:28:05.149127 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:05.149054    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-qtdvz_21a2db47-94e4-4d2b-9cb2-0b2bca64ef5b/dns-node-resolver/0.log"
Sep 16 17:28:05.349643 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:05.349617    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-99d9f4-g4nhp_7a9b26b4-3f5f-40d2-b866-ad3915e771bb/router/0.log"
Sep 16 17:28:05.948258 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:05.948206    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-t8qph_7886f4b3-cb86-4a0e-ae6c-809ac7a567fa/serve-healthcheck-canary/0.log"
Sep 16 17:28:06.099099 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:06.099071    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9ba8e716-8f2b-4fe4-a140-707f7335fbef" path="/var/lib/kubelet/pods/9ba8e716-8f2b-4fe4-a140-707f7335fbef/volumes"
Sep 16 17:28:13.975886 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:13.975804    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-d77b4b4b-2f2bv" podUID="3c32493a-a573-4c14-9e85-9f0faef07cd5" containerName="console" containerID="cri-o://e02951a32900a7648555dbe0a485791c10f0ae2d2320aa3c9a0b424ecba12261" gracePeriod=15
Sep 16 17:28:14.269916 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:14.269888    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-d77b4b4b-2f2bv"
Sep 16 17:28:14.346697 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:14.346552    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3c32493a-a573-4c14-9e85-9f0faef07cd5-service-ca\") pod \"3c32493a-a573-4c14-9e85-9f0faef07cd5\" (UID: \"3c32493a-a573-4c14-9e85-9f0faef07cd5\") "
Sep 16 17:28:14.346697 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:14.346608    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3c32493a-a573-4c14-9e85-9f0faef07cd5-console-config\") pod \"3c32493a-a573-4c14-9e85-9f0faef07cd5\" (UID: \"3c32493a-a573-4c14-9e85-9f0faef07cd5\") "
Sep 16 17:28:14.346992 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:14.346943    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c32493a-a573-4c14-9e85-9f0faef07cd5-service-ca" (OuterVolumeSpecName: "service-ca") pod "3c32493a-a573-4c14-9e85-9f0faef07cd5" (UID: "3c32493a-a573-4c14-9e85-9f0faef07cd5"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 17:28:14.346992 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:14.346948    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c32493a-a573-4c14-9e85-9f0faef07cd5-console-config" (OuterVolumeSpecName: "console-config") pod "3c32493a-a573-4c14-9e85-9f0faef07cd5" (UID: "3c32493a-a573-4c14-9e85-9f0faef07cd5"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 17:28:14.346992 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:14.346978    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3c32493a-a573-4c14-9e85-9f0faef07cd5-console-serving-cert\") pod \"3c32493a-a573-4c14-9e85-9f0faef07cd5\" (UID: \"3c32493a-a573-4c14-9e85-9f0faef07cd5\") "
Sep 16 17:28:14.347161 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:14.347015    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3c32493a-a573-4c14-9e85-9f0faef07cd5-trusted-ca-bundle\") pod \"3c32493a-a573-4c14-9e85-9f0faef07cd5\" (UID: \"3c32493a-a573-4c14-9e85-9f0faef07cd5\") "
Sep 16 17:28:14.347161 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:14.347043    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zgh8f\" (UniqueName: \"kubernetes.io/projected/3c32493a-a573-4c14-9e85-9f0faef07cd5-kube-api-access-zgh8f\") pod \"3c32493a-a573-4c14-9e85-9f0faef07cd5\" (UID: \"3c32493a-a573-4c14-9e85-9f0faef07cd5\") "
Sep 16 17:28:14.347161 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:14.347079    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3c32493a-a573-4c14-9e85-9f0faef07cd5-console-oauth-config\") pod \"3c32493a-a573-4c14-9e85-9f0faef07cd5\" (UID: \"3c32493a-a573-4c14-9e85-9f0faef07cd5\") "
Sep 16 17:28:14.347161 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:14.347124    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3c32493a-a573-4c14-9e85-9f0faef07cd5-oauth-serving-cert\") pod \"3c32493a-a573-4c14-9e85-9f0faef07cd5\" (UID: \"3c32493a-a573-4c14-9e85-9f0faef07cd5\") "
Sep 16 17:28:14.347963 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:14.347401    2579 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3c32493a-a573-4c14-9e85-9f0faef07cd5-service-ca\") on node \"ip-10-0-138-73.ec2.internal\" DevicePath \"\""
Sep 16 17:28:14.347963 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:14.347424    2579 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3c32493a-a573-4c14-9e85-9f0faef07cd5-console-config\") on node \"ip-10-0-138-73.ec2.internal\" DevicePath \"\""
Sep 16 17:28:14.347963 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:14.347668    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c32493a-a573-4c14-9e85-9f0faef07cd5-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "3c32493a-a573-4c14-9e85-9f0faef07cd5" (UID: "3c32493a-a573-4c14-9e85-9f0faef07cd5"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 17:28:14.348207 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:14.347976    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c32493a-a573-4c14-9e85-9f0faef07cd5-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "3c32493a-a573-4c14-9e85-9f0faef07cd5" (UID: "3c32493a-a573-4c14-9e85-9f0faef07cd5"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 17:28:14.349533 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:14.349504    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c32493a-a573-4c14-9e85-9f0faef07cd5-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "3c32493a-a573-4c14-9e85-9f0faef07cd5" (UID: "3c32493a-a573-4c14-9e85-9f0faef07cd5"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 17:28:14.349644 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:14.349614    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c32493a-a573-4c14-9e85-9f0faef07cd5-kube-api-access-zgh8f" (OuterVolumeSpecName: "kube-api-access-zgh8f") pod "3c32493a-a573-4c14-9e85-9f0faef07cd5" (UID: "3c32493a-a573-4c14-9e85-9f0faef07cd5"). InnerVolumeSpecName "kube-api-access-zgh8f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 17:28:14.349829 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:14.349807    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c32493a-a573-4c14-9e85-9f0faef07cd5-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "3c32493a-a573-4c14-9e85-9f0faef07cd5" (UID: "3c32493a-a573-4c14-9e85-9f0faef07cd5"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 17:28:14.448918 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:14.448888    2579 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3c32493a-a573-4c14-9e85-9f0faef07cd5-console-serving-cert\") on node \"ip-10-0-138-73.ec2.internal\" DevicePath \"\""
Sep 16 17:28:14.448918 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:14.448915    2579 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3c32493a-a573-4c14-9e85-9f0faef07cd5-trusted-ca-bundle\") on node \"ip-10-0-138-73.ec2.internal\" DevicePath \"\""
Sep 16 17:28:14.448918 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:14.448924    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zgh8f\" (UniqueName: \"kubernetes.io/projected/3c32493a-a573-4c14-9e85-9f0faef07cd5-kube-api-access-zgh8f\") on node \"ip-10-0-138-73.ec2.internal\" DevicePath \"\""
Sep 16 17:28:14.449141 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:14.448933    2579 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3c32493a-a573-4c14-9e85-9f0faef07cd5-console-oauth-config\") on node \"ip-10-0-138-73.ec2.internal\" DevicePath \"\""
Sep 16 17:28:14.449141 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:14.448943    2579 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3c32493a-a573-4c14-9e85-9f0faef07cd5-oauth-serving-cert\") on node \"ip-10-0-138-73.ec2.internal\" DevicePath \"\""
Sep 16 17:28:14.929706 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:14.929667    2579 generic.go:356] "Generic (PLEG): container finished" podID="3c32493a-a573-4c14-9e85-9f0faef07cd5" containerID="e02951a32900a7648555dbe0a485791c10f0ae2d2320aa3c9a0b424ecba12261" exitCode=0
Sep 16 17:28:14.929706 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:14.929706    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-d77b4b4b-2f2bv"
Sep 16 17:28:14.929953 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:14.929729    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-d77b4b4b-2f2bv" event={"ID":"3c32493a-a573-4c14-9e85-9f0faef07cd5","Type":"ContainerDied","Data":"e02951a32900a7648555dbe0a485791c10f0ae2d2320aa3c9a0b424ecba12261"}
Sep 16 17:28:14.929953 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:14.929782    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-d77b4b4b-2f2bv" event={"ID":"3c32493a-a573-4c14-9e85-9f0faef07cd5","Type":"ContainerDied","Data":"da4421835201ef23e8eab2beb4acd37485f0e1271cfe243224cdc849fe889b60"}
Sep 16 17:28:14.929953 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:14.929811    2579 scope.go:117] "RemoveContainer" containerID="e02951a32900a7648555dbe0a485791c10f0ae2d2320aa3c9a0b424ecba12261"
Sep 16 17:28:14.939200 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:14.939175    2579 scope.go:117] "RemoveContainer" containerID="e02951a32900a7648555dbe0a485791c10f0ae2d2320aa3c9a0b424ecba12261"
Sep 16 17:28:14.939507 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:28:14.939482    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e02951a32900a7648555dbe0a485791c10f0ae2d2320aa3c9a0b424ecba12261\": container with ID starting with e02951a32900a7648555dbe0a485791c10f0ae2d2320aa3c9a0b424ecba12261 not found: ID does not exist" containerID="e02951a32900a7648555dbe0a485791c10f0ae2d2320aa3c9a0b424ecba12261"
Sep 16 17:28:14.939575 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:14.939513    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e02951a32900a7648555dbe0a485791c10f0ae2d2320aa3c9a0b424ecba12261"} err="failed to get container status \"e02951a32900a7648555dbe0a485791c10f0ae2d2320aa3c9a0b424ecba12261\": rpc error: code = NotFound desc = could not find container \"e02951a32900a7648555dbe0a485791c10f0ae2d2320aa3c9a0b424ecba12261\": container with ID starting with e02951a32900a7648555dbe0a485791c10f0ae2d2320aa3c9a0b424ecba12261 not found: ID does not exist"
Sep 16 17:28:14.952874 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:14.952832    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-d77b4b4b-2f2bv"]
Sep 16 17:28:14.957310 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:14.957286    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-d77b4b4b-2f2bv"]
Sep 16 17:28:16.099571 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:16.099533    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3c32493a-a573-4c14-9e85-9f0faef07cd5" path="/var/lib/kubelet/pods/3c32493a-a573-4c14-9e85-9f0faef07cd5/volumes"
Sep 16 17:28:17.451706 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:17.451674    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-5b8b89bf95-slsg6"
Sep 16 17:28:17.455646 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:17.455624    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-5b8b89bf95-slsg6"
Sep 16 17:28:22.967072 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:22.967030    2579 generic.go:356] "Generic (PLEG): container finished" podID="4cf3f89a-3a7d-4651-94c3-26fe3ee314cc" containerID="f10827809f462359605a2ce4fce62103c365c65855b44f6da745afbac68a1398" exitCode=0
Sep 16 17:28:22.967537 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:22.967104    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-7v4db" event={"ID":"4cf3f89a-3a7d-4651-94c3-26fe3ee314cc","Type":"ContainerDied","Data":"f10827809f462359605a2ce4fce62103c365c65855b44f6da745afbac68a1398"}
Sep 16 17:28:22.967537 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:22.967494    2579 scope.go:117] "RemoveContainer" containerID="f10827809f462359605a2ce4fce62103c365c65855b44f6da745afbac68a1398"
Sep 16 17:28:23.974072 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:23.974037    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-7v4db" event={"ID":"4cf3f89a-3a7d-4651-94c3-26fe3ee314cc","Type":"ContainerStarted","Data":"4d6c75ea35093194ea741a7c0132227bbfb18da7a3bb3452e16ba0f90bf1b8b7"}
Sep 16 17:28:53.132027 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:53.131991    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 17:28:53.132466 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:53.132265    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="a2ad2862-5d15-46b2-aaeb-a96ad6af3255" containerName="alertmanager" containerID="cri-o://6433d2236aad85b04301e28bdad7e4af1109f4cb9ac844bc47af43a367573ba9" gracePeriod=120
Sep 16 17:28:53.132466 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:53.132332    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="a2ad2862-5d15-46b2-aaeb-a96ad6af3255" containerName="kube-rbac-proxy-metric" containerID="cri-o://915c986887654ba3ce50062e4938339cb3504d0da594ebd5ad60868f60b7c526" gracePeriod=120
Sep 16 17:28:53.132466 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:53.132371    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="a2ad2862-5d15-46b2-aaeb-a96ad6af3255" containerName="prom-label-proxy" containerID="cri-o://d68bec048ec2edc292dd1029ceba90f9feb844152cf756ad0de79d86c8636422" gracePeriod=120
Sep 16 17:28:53.132466 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:53.132390    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="a2ad2862-5d15-46b2-aaeb-a96ad6af3255" containerName="kube-rbac-proxy-web" containerID="cri-o://db923488d4f7ebd57a238df7f99586eefd464ab5be7c5284676df6b897c3b734" gracePeriod=120
Sep 16 17:28:53.132466 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:53.132417    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="a2ad2862-5d15-46b2-aaeb-a96ad6af3255" containerName="config-reloader" containerID="cri-o://7154276c8a3fdc2fc102e0015c16c36591e850d44a73305fcf9d4514c616d25f" gracePeriod=120
Sep 16 17:28:53.132671 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:53.132401    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="a2ad2862-5d15-46b2-aaeb-a96ad6af3255" containerName="kube-rbac-proxy" containerID="cri-o://84faa18e8faf36d1f969d8b0e7cd3ab717e604a6c2ae21561e670c53dfe1613f" gracePeriod=120
Sep 16 17:28:54.091899 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:54.091865    2579 generic.go:356] "Generic (PLEG): container finished" podID="a2ad2862-5d15-46b2-aaeb-a96ad6af3255" containerID="d68bec048ec2edc292dd1029ceba90f9feb844152cf756ad0de79d86c8636422" exitCode=0
Sep 16 17:28:54.091899 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:54.091879    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a2ad2862-5d15-46b2-aaeb-a96ad6af3255","Type":"ContainerDied","Data":"d68bec048ec2edc292dd1029ceba90f9feb844152cf756ad0de79d86c8636422"}
Sep 16 17:28:54.091899 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:54.091894    2579 generic.go:356] "Generic (PLEG): container finished" podID="a2ad2862-5d15-46b2-aaeb-a96ad6af3255" containerID="84faa18e8faf36d1f969d8b0e7cd3ab717e604a6c2ae21561e670c53dfe1613f" exitCode=0
Sep 16 17:28:54.091899 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:54.091903    2579 generic.go:356] "Generic (PLEG): container finished" podID="a2ad2862-5d15-46b2-aaeb-a96ad6af3255" containerID="7154276c8a3fdc2fc102e0015c16c36591e850d44a73305fcf9d4514c616d25f" exitCode=0
Sep 16 17:28:54.091899 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:54.091909    2579 generic.go:356] "Generic (PLEG): container finished" podID="a2ad2862-5d15-46b2-aaeb-a96ad6af3255" containerID="6433d2236aad85b04301e28bdad7e4af1109f4cb9ac844bc47af43a367573ba9" exitCode=0
Sep 16 17:28:54.092198 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:54.091945    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a2ad2862-5d15-46b2-aaeb-a96ad6af3255","Type":"ContainerDied","Data":"84faa18e8faf36d1f969d8b0e7cd3ab717e604a6c2ae21561e670c53dfe1613f"}
Sep 16 17:28:54.092198 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:54.091973    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a2ad2862-5d15-46b2-aaeb-a96ad6af3255","Type":"ContainerDied","Data":"7154276c8a3fdc2fc102e0015c16c36591e850d44a73305fcf9d4514c616d25f"}
Sep 16 17:28:54.092198 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:54.091999    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a2ad2862-5d15-46b2-aaeb-a96ad6af3255","Type":"ContainerDied","Data":"6433d2236aad85b04301e28bdad7e4af1109f4cb9ac844bc47af43a367573ba9"}
Sep 16 17:28:54.377914 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:54.377886    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 16 17:28:54.487895 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:54.487836    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/a2ad2862-5d15-46b2-aaeb-a96ad6af3255-alertmanager-main-db\") pod \"a2ad2862-5d15-46b2-aaeb-a96ad6af3255\" (UID: \"a2ad2862-5d15-46b2-aaeb-a96ad6af3255\") "
Sep 16 17:28:54.488076 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:54.487906    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a2ad2862-5d15-46b2-aaeb-a96ad6af3255-secret-alertmanager-kube-rbac-proxy-web\") pod \"a2ad2862-5d15-46b2-aaeb-a96ad6af3255\" (UID: \"a2ad2862-5d15-46b2-aaeb-a96ad6af3255\") "
Sep 16 17:28:54.488076 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:54.487933    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a2ad2862-5d15-46b2-aaeb-a96ad6af3255-secret-alertmanager-kube-rbac-proxy\") pod \"a2ad2862-5d15-46b2-aaeb-a96ad6af3255\" (UID: \"a2ad2862-5d15-46b2-aaeb-a96ad6af3255\") "
Sep 16 17:28:54.488076 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:54.487972    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a2ad2862-5d15-46b2-aaeb-a96ad6af3255-config-volume\") pod \"a2ad2862-5d15-46b2-aaeb-a96ad6af3255\" (UID: \"a2ad2862-5d15-46b2-aaeb-a96ad6af3255\") "
Sep 16 17:28:54.488076 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:54.488020    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a2ad2862-5d15-46b2-aaeb-a96ad6af3255-cluster-tls-config\") pod \"a2ad2862-5d15-46b2-aaeb-a96ad6af3255\" (UID: \"a2ad2862-5d15-46b2-aaeb-a96ad6af3255\") "
Sep 16 17:28:54.488076 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:54.488057    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a2ad2862-5d15-46b2-aaeb-a96ad6af3255-alertmanager-trusted-ca-bundle\") pod \"a2ad2862-5d15-46b2-aaeb-a96ad6af3255\" (UID: \"a2ad2862-5d15-46b2-aaeb-a96ad6af3255\") "
Sep 16 17:28:54.488332 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:54.488094    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-blprt\" (UniqueName: \"kubernetes.io/projected/a2ad2862-5d15-46b2-aaeb-a96ad6af3255-kube-api-access-blprt\") pod \"a2ad2862-5d15-46b2-aaeb-a96ad6af3255\" (UID: \"a2ad2862-5d15-46b2-aaeb-a96ad6af3255\") "
Sep 16 17:28:54.488332 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:54.488129    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a2ad2862-5d15-46b2-aaeb-a96ad6af3255-metrics-client-ca\") pod \"a2ad2862-5d15-46b2-aaeb-a96ad6af3255\" (UID: \"a2ad2862-5d15-46b2-aaeb-a96ad6af3255\") "
Sep 16 17:28:54.488332 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:54.488161    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2ad2862-5d15-46b2-aaeb-a96ad6af3255-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "a2ad2862-5d15-46b2-aaeb-a96ad6af3255" (UID: "a2ad2862-5d15-46b2-aaeb-a96ad6af3255"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:28:54.488332 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:54.488184    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/a2ad2862-5d15-46b2-aaeb-a96ad6af3255-secret-alertmanager-kube-rbac-proxy-metric\") pod \"a2ad2862-5d15-46b2-aaeb-a96ad6af3255\" (UID: \"a2ad2862-5d15-46b2-aaeb-a96ad6af3255\") "
Sep 16 17:28:54.488332 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:54.488246    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a2ad2862-5d15-46b2-aaeb-a96ad6af3255-config-out\") pod \"a2ad2862-5d15-46b2-aaeb-a96ad6af3255\" (UID: \"a2ad2862-5d15-46b2-aaeb-a96ad6af3255\") "
Sep 16 17:28:54.488332 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:54.488276    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a2ad2862-5d15-46b2-aaeb-a96ad6af3255-web-config\") pod \"a2ad2862-5d15-46b2-aaeb-a96ad6af3255\" (UID: \"a2ad2862-5d15-46b2-aaeb-a96ad6af3255\") "
Sep 16 17:28:54.488332 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:54.488312    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a2ad2862-5d15-46b2-aaeb-a96ad6af3255-secret-alertmanager-main-tls\") pod \"a2ad2862-5d15-46b2-aaeb-a96ad6af3255\" (UID: \"a2ad2862-5d15-46b2-aaeb-a96ad6af3255\") "
Sep 16 17:28:54.488742 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:54.488345    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a2ad2862-5d15-46b2-aaeb-a96ad6af3255-tls-assets\") pod \"a2ad2862-5d15-46b2-aaeb-a96ad6af3255\" (UID: \"a2ad2862-5d15-46b2-aaeb-a96ad6af3255\") "
Sep 16 17:28:54.488742 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:54.488493    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2ad2862-5d15-46b2-aaeb-a96ad6af3255-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "a2ad2862-5d15-46b2-aaeb-a96ad6af3255" (UID: "a2ad2862-5d15-46b2-aaeb-a96ad6af3255"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 17:28:54.488742 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:54.488609    2579 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a2ad2862-5d15-46b2-aaeb-a96ad6af3255-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-138-73.ec2.internal\" DevicePath \"\""
Sep 16 17:28:54.488742 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:54.488626    2579 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/a2ad2862-5d15-46b2-aaeb-a96ad6af3255-alertmanager-main-db\") on node \"ip-10-0-138-73.ec2.internal\" DevicePath \"\""
Sep 16 17:28:54.489422 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:54.489360    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2ad2862-5d15-46b2-aaeb-a96ad6af3255-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "a2ad2862-5d15-46b2-aaeb-a96ad6af3255" (UID: "a2ad2862-5d15-46b2-aaeb-a96ad6af3255"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 17:28:54.490878 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:54.490814    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2ad2862-5d15-46b2-aaeb-a96ad6af3255-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "a2ad2862-5d15-46b2-aaeb-a96ad6af3255" (UID: "a2ad2862-5d15-46b2-aaeb-a96ad6af3255"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 17:28:54.491411 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:54.491388    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2ad2862-5d15-46b2-aaeb-a96ad6af3255-kube-api-access-blprt" (OuterVolumeSpecName: "kube-api-access-blprt") pod "a2ad2862-5d15-46b2-aaeb-a96ad6af3255" (UID: "a2ad2862-5d15-46b2-aaeb-a96ad6af3255"). InnerVolumeSpecName "kube-api-access-blprt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 17:28:54.491496 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:54.491439    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2ad2862-5d15-46b2-aaeb-a96ad6af3255-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "a2ad2862-5d15-46b2-aaeb-a96ad6af3255" (UID: "a2ad2862-5d15-46b2-aaeb-a96ad6af3255"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 17:28:54.491599 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:54.491573    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2ad2862-5d15-46b2-aaeb-a96ad6af3255-config-out" (OuterVolumeSpecName: "config-out") pod "a2ad2862-5d15-46b2-aaeb-a96ad6af3255" (UID: "a2ad2862-5d15-46b2-aaeb-a96ad6af3255"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:28:54.492020 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:54.491994    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2ad2862-5d15-46b2-aaeb-a96ad6af3255-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "a2ad2862-5d15-46b2-aaeb-a96ad6af3255" (UID: "a2ad2862-5d15-46b2-aaeb-a96ad6af3255"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 17:28:54.492098 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:54.491997    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2ad2862-5d15-46b2-aaeb-a96ad6af3255-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "a2ad2862-5d15-46b2-aaeb-a96ad6af3255" (UID: "a2ad2862-5d15-46b2-aaeb-a96ad6af3255"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 17:28:54.492160 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:54.492139    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2ad2862-5d15-46b2-aaeb-a96ad6af3255-config-volume" (OuterVolumeSpecName: "config-volume") pod "a2ad2862-5d15-46b2-aaeb-a96ad6af3255" (UID: "a2ad2862-5d15-46b2-aaeb-a96ad6af3255"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 17:28:54.492753 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:54.492724    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2ad2862-5d15-46b2-aaeb-a96ad6af3255-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "a2ad2862-5d15-46b2-aaeb-a96ad6af3255" (UID: "a2ad2862-5d15-46b2-aaeb-a96ad6af3255"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 17:28:54.495348 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:54.495320    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2ad2862-5d15-46b2-aaeb-a96ad6af3255-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "a2ad2862-5d15-46b2-aaeb-a96ad6af3255" (UID: "a2ad2862-5d15-46b2-aaeb-a96ad6af3255"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 17:28:54.502499 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:54.502477    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2ad2862-5d15-46b2-aaeb-a96ad6af3255-web-config" (OuterVolumeSpecName: "web-config") pod "a2ad2862-5d15-46b2-aaeb-a96ad6af3255" (UID: "a2ad2862-5d15-46b2-aaeb-a96ad6af3255"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 17:28:54.589644 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:54.589614    2579 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a2ad2862-5d15-46b2-aaeb-a96ad6af3255-cluster-tls-config\") on node \"ip-10-0-138-73.ec2.internal\" DevicePath \"\""
Sep 16 17:28:54.589644 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:54.589640    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-blprt\" (UniqueName: \"kubernetes.io/projected/a2ad2862-5d15-46b2-aaeb-a96ad6af3255-kube-api-access-blprt\") on node \"ip-10-0-138-73.ec2.internal\" DevicePath \"\""
Sep 16 17:28:54.589644 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:54.589651    2579 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a2ad2862-5d15-46b2-aaeb-a96ad6af3255-metrics-client-ca\") on node \"ip-10-0-138-73.ec2.internal\" DevicePath \"\""
Sep 16 17:28:54.589838 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:54.589661    2579 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/a2ad2862-5d15-46b2-aaeb-a96ad6af3255-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-138-73.ec2.internal\" DevicePath \"\""
Sep 16 17:28:54.589838 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:54.589671    2579 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a2ad2862-5d15-46b2-aaeb-a96ad6af3255-config-out\") on node \"ip-10-0-138-73.ec2.internal\" DevicePath \"\""
Sep 16 17:28:54.589838 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:54.589678    2579 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a2ad2862-5d15-46b2-aaeb-a96ad6af3255-web-config\") on node \"ip-10-0-138-73.ec2.internal\" DevicePath \"\""
Sep 16 17:28:54.589838 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:54.589686    2579 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a2ad2862-5d15-46b2-aaeb-a96ad6af3255-secret-alertmanager-main-tls\") on node \"ip-10-0-138-73.ec2.internal\" DevicePath \"\""
Sep 16 17:28:54.589838 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:54.589693    2579 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a2ad2862-5d15-46b2-aaeb-a96ad6af3255-tls-assets\") on node \"ip-10-0-138-73.ec2.internal\" DevicePath \"\""
Sep 16 17:28:54.589838 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:54.589702    2579 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a2ad2862-5d15-46b2-aaeb-a96ad6af3255-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-138-73.ec2.internal\" DevicePath \"\""
Sep 16 17:28:54.589838 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:54.589710    2579 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a2ad2862-5d15-46b2-aaeb-a96ad6af3255-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-138-73.ec2.internal\" DevicePath \"\""
Sep 16 17:28:54.589838 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:54.589719    2579 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a2ad2862-5d15-46b2-aaeb-a96ad6af3255-config-volume\") on node \"ip-10-0-138-73.ec2.internal\" DevicePath \"\""
Sep 16 17:28:55.099029 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:55.099000    2579 generic.go:356] "Generic (PLEG): container finished" podID="a2ad2862-5d15-46b2-aaeb-a96ad6af3255" containerID="915c986887654ba3ce50062e4938339cb3504d0da594ebd5ad60868f60b7c526" exitCode=0
Sep 16 17:28:55.099029 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:55.099022    2579 generic.go:356] "Generic (PLEG): container finished" podID="a2ad2862-5d15-46b2-aaeb-a96ad6af3255" containerID="db923488d4f7ebd57a238df7f99586eefd464ab5be7c5284676df6b897c3b734" exitCode=0
Sep 16 17:28:55.099227 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:55.099072    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 16 17:28:55.099227 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:55.099084    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a2ad2862-5d15-46b2-aaeb-a96ad6af3255","Type":"ContainerDied","Data":"915c986887654ba3ce50062e4938339cb3504d0da594ebd5ad60868f60b7c526"}
Sep 16 17:28:55.099227 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:55.099142    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a2ad2862-5d15-46b2-aaeb-a96ad6af3255","Type":"ContainerDied","Data":"db923488d4f7ebd57a238df7f99586eefd464ab5be7c5284676df6b897c3b734"}
Sep 16 17:28:55.099227 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:55.099191    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a2ad2862-5d15-46b2-aaeb-a96ad6af3255","Type":"ContainerDied","Data":"eeb5ab817b93d97099c1c9ab181acf794ddc2ffa99d2b22d5edccb2c1fbc3d17"}
Sep 16 17:28:55.099227 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:55.099210    2579 scope.go:117] "RemoveContainer" containerID="d68bec048ec2edc292dd1029ceba90f9feb844152cf756ad0de79d86c8636422"
Sep 16 17:28:55.106875 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:55.106829    2579 scope.go:117] "RemoveContainer" containerID="915c986887654ba3ce50062e4938339cb3504d0da594ebd5ad60868f60b7c526"
Sep 16 17:28:55.117492 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:55.117464    2579 scope.go:117] "RemoveContainer" containerID="84faa18e8faf36d1f969d8b0e7cd3ab717e604a6c2ae21561e670c53dfe1613f"
Sep 16 17:28:55.124545 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:55.124524    2579 scope.go:117] "RemoveContainer" containerID="db923488d4f7ebd57a238df7f99586eefd464ab5be7c5284676df6b897c3b734"
Sep 16 17:28:55.128444 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:55.128422    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 17:28:55.134069 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:55.134048    2579 scope.go:117] "RemoveContainer" containerID="7154276c8a3fdc2fc102e0015c16c36591e850d44a73305fcf9d4514c616d25f"
Sep 16 17:28:55.136047 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:55.136027    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 17:28:55.141201 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:55.141184    2579 scope.go:117] "RemoveContainer" containerID="6433d2236aad85b04301e28bdad7e4af1109f4cb9ac844bc47af43a367573ba9"
Sep 16 17:28:55.148123 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:55.148103    2579 scope.go:117] "RemoveContainer" containerID="8bc46bc5b55389fcb60ab6a68987c6fdc2ca9ce5cb68aad0b86df70da96c7da1"
Sep 16 17:28:55.155046 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:55.155031    2579 scope.go:117] "RemoveContainer" containerID="d68bec048ec2edc292dd1029ceba90f9feb844152cf756ad0de79d86c8636422"
Sep 16 17:28:55.155310 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:28:55.155287    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d68bec048ec2edc292dd1029ceba90f9feb844152cf756ad0de79d86c8636422\": container with ID starting with d68bec048ec2edc292dd1029ceba90f9feb844152cf756ad0de79d86c8636422 not found: ID does not exist" containerID="d68bec048ec2edc292dd1029ceba90f9feb844152cf756ad0de79d86c8636422"
Sep 16 17:28:55.155359 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:55.155313    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d68bec048ec2edc292dd1029ceba90f9feb844152cf756ad0de79d86c8636422"} err="failed to get container status \"d68bec048ec2edc292dd1029ceba90f9feb844152cf756ad0de79d86c8636422\": rpc error: code = NotFound desc = could not find container \"d68bec048ec2edc292dd1029ceba90f9feb844152cf756ad0de79d86c8636422\": container with ID starting with d68bec048ec2edc292dd1029ceba90f9feb844152cf756ad0de79d86c8636422 not found: ID does not exist"
Sep 16 17:28:55.155359 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:55.155331    2579 scope.go:117] "RemoveContainer" containerID="915c986887654ba3ce50062e4938339cb3504d0da594ebd5ad60868f60b7c526"
Sep 16 17:28:55.155581 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:28:55.155564    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"915c986887654ba3ce50062e4938339cb3504d0da594ebd5ad60868f60b7c526\": container with ID starting with 915c986887654ba3ce50062e4938339cb3504d0da594ebd5ad60868f60b7c526 not found: ID does not exist" containerID="915c986887654ba3ce50062e4938339cb3504d0da594ebd5ad60868f60b7c526"
Sep 16 17:28:55.155617 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:55.155596    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"915c986887654ba3ce50062e4938339cb3504d0da594ebd5ad60868f60b7c526"} err="failed to get container status \"915c986887654ba3ce50062e4938339cb3504d0da594ebd5ad60868f60b7c526\": rpc error: code = NotFound desc = could not find container \"915c986887654ba3ce50062e4938339cb3504d0da594ebd5ad60868f60b7c526\": container with ID starting with 915c986887654ba3ce50062e4938339cb3504d0da594ebd5ad60868f60b7c526 not found: ID does not exist"
Sep 16 17:28:55.155617 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:55.155613    2579 scope.go:117] "RemoveContainer" containerID="84faa18e8faf36d1f969d8b0e7cd3ab717e604a6c2ae21561e670c53dfe1613f"
Sep 16 17:28:55.155837 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:28:55.155823    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84faa18e8faf36d1f969d8b0e7cd3ab717e604a6c2ae21561e670c53dfe1613f\": container with ID starting with 84faa18e8faf36d1f969d8b0e7cd3ab717e604a6c2ae21561e670c53dfe1613f not found: ID does not exist" containerID="84faa18e8faf36d1f969d8b0e7cd3ab717e604a6c2ae21561e670c53dfe1613f"
Sep 16 17:28:55.155961 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:55.155854    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84faa18e8faf36d1f969d8b0e7cd3ab717e604a6c2ae21561e670c53dfe1613f"} err="failed to get container status \"84faa18e8faf36d1f969d8b0e7cd3ab717e604a6c2ae21561e670c53dfe1613f\": rpc error: code = NotFound desc = could not find container \"84faa18e8faf36d1f969d8b0e7cd3ab717e604a6c2ae21561e670c53dfe1613f\": container with ID starting with 84faa18e8faf36d1f969d8b0e7cd3ab717e604a6c2ae21561e670c53dfe1613f not found: ID does not exist"
Sep 16 17:28:55.155961 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:55.155868    2579 scope.go:117] "RemoveContainer" containerID="db923488d4f7ebd57a238df7f99586eefd464ab5be7c5284676df6b897c3b734"
Sep 16 17:28:55.156092 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:28:55.156077    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db923488d4f7ebd57a238df7f99586eefd464ab5be7c5284676df6b897c3b734\": container with ID starting with db923488d4f7ebd57a238df7f99586eefd464ab5be7c5284676df6b897c3b734 not found: ID does not exist" containerID="db923488d4f7ebd57a238df7f99586eefd464ab5be7c5284676df6b897c3b734"
Sep 16 17:28:55.156127 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:55.156094    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db923488d4f7ebd57a238df7f99586eefd464ab5be7c5284676df6b897c3b734"} err="failed to get container status \"db923488d4f7ebd57a238df7f99586eefd464ab5be7c5284676df6b897c3b734\": rpc error: code = NotFound desc = could not find container \"db923488d4f7ebd57a238df7f99586eefd464ab5be7c5284676df6b897c3b734\": container with ID starting with db923488d4f7ebd57a238df7f99586eefd464ab5be7c5284676df6b897c3b734 not found: ID does not exist"
Sep 16 17:28:55.156127 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:55.156105    2579 scope.go:117] "RemoveContainer" containerID="7154276c8a3fdc2fc102e0015c16c36591e850d44a73305fcf9d4514c616d25f"
Sep 16 17:28:55.156366 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:28:55.156348    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7154276c8a3fdc2fc102e0015c16c36591e850d44a73305fcf9d4514c616d25f\": container with ID starting with 7154276c8a3fdc2fc102e0015c16c36591e850d44a73305fcf9d4514c616d25f not found: ID does not exist" containerID="7154276c8a3fdc2fc102e0015c16c36591e850d44a73305fcf9d4514c616d25f"
Sep 16 17:28:55.156413 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:55.156370    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7154276c8a3fdc2fc102e0015c16c36591e850d44a73305fcf9d4514c616d25f"} err="failed to get container status \"7154276c8a3fdc2fc102e0015c16c36591e850d44a73305fcf9d4514c616d25f\": rpc error: code = NotFound desc = could not find container \"7154276c8a3fdc2fc102e0015c16c36591e850d44a73305fcf9d4514c616d25f\": container with ID starting with 7154276c8a3fdc2fc102e0015c16c36591e850d44a73305fcf9d4514c616d25f not found: ID does not exist"
Sep 16 17:28:55.156413 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:55.156382    2579 scope.go:117] "RemoveContainer" containerID="6433d2236aad85b04301e28bdad7e4af1109f4cb9ac844bc47af43a367573ba9"
Sep 16 17:28:55.156646 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:28:55.156630    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6433d2236aad85b04301e28bdad7e4af1109f4cb9ac844bc47af43a367573ba9\": container with ID starting with 6433d2236aad85b04301e28bdad7e4af1109f4cb9ac844bc47af43a367573ba9 not found: ID does not exist" containerID="6433d2236aad85b04301e28bdad7e4af1109f4cb9ac844bc47af43a367573ba9"
Sep 16 17:28:55.156692 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:55.156649    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6433d2236aad85b04301e28bdad7e4af1109f4cb9ac844bc47af43a367573ba9"} err="failed to get container status \"6433d2236aad85b04301e28bdad7e4af1109f4cb9ac844bc47af43a367573ba9\": rpc error: code = NotFound desc = could not find container \"6433d2236aad85b04301e28bdad7e4af1109f4cb9ac844bc47af43a367573ba9\": container with ID starting with 6433d2236aad85b04301e28bdad7e4af1109f4cb9ac844bc47af43a367573ba9 not found: ID does not exist"
Sep 16 17:28:55.156692 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:55.156662    2579 scope.go:117] "RemoveContainer" containerID="8bc46bc5b55389fcb60ab6a68987c6fdc2ca9ce5cb68aad0b86df70da96c7da1"
Sep 16 17:28:55.156904 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:28:55.156884    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8bc46bc5b55389fcb60ab6a68987c6fdc2ca9ce5cb68aad0b86df70da96c7da1\": container with ID starting with 8bc46bc5b55389fcb60ab6a68987c6fdc2ca9ce5cb68aad0b86df70da96c7da1 not found: ID does not exist" containerID="8bc46bc5b55389fcb60ab6a68987c6fdc2ca9ce5cb68aad0b86df70da96c7da1"
Sep 16 17:28:55.156944 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:55.156911    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8bc46bc5b55389fcb60ab6a68987c6fdc2ca9ce5cb68aad0b86df70da96c7da1"} err="failed to get container status \"8bc46bc5b55389fcb60ab6a68987c6fdc2ca9ce5cb68aad0b86df70da96c7da1\": rpc error: code = NotFound desc = could not find container \"8bc46bc5b55389fcb60ab6a68987c6fdc2ca9ce5cb68aad0b86df70da96c7da1\": container with ID starting with 8bc46bc5b55389fcb60ab6a68987c6fdc2ca9ce5cb68aad0b86df70da96c7da1 not found: ID does not exist"
Sep 16 17:28:55.156944 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:55.156925    2579 scope.go:117] "RemoveContainer" containerID="d68bec048ec2edc292dd1029ceba90f9feb844152cf756ad0de79d86c8636422"
Sep 16 17:28:55.157146 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:55.157129    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d68bec048ec2edc292dd1029ceba90f9feb844152cf756ad0de79d86c8636422"} err="failed to get container status \"d68bec048ec2edc292dd1029ceba90f9feb844152cf756ad0de79d86c8636422\": rpc error: code = NotFound desc = could not find container \"d68bec048ec2edc292dd1029ceba90f9feb844152cf756ad0de79d86c8636422\": container with ID starting with d68bec048ec2edc292dd1029ceba90f9feb844152cf756ad0de79d86c8636422 not found: ID does not exist"
Sep 16 17:28:55.157182 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:55.157147    2579 scope.go:117] "RemoveContainer" containerID="915c986887654ba3ce50062e4938339cb3504d0da594ebd5ad60868f60b7c526"
Sep 16 17:28:55.157351 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:55.157336    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"915c986887654ba3ce50062e4938339cb3504d0da594ebd5ad60868f60b7c526"} err="failed to get container status \"915c986887654ba3ce50062e4938339cb3504d0da594ebd5ad60868f60b7c526\": rpc error: code = NotFound desc = could not find container \"915c986887654ba3ce50062e4938339cb3504d0da594ebd5ad60868f60b7c526\": container with ID starting with 915c986887654ba3ce50062e4938339cb3504d0da594ebd5ad60868f60b7c526 not found: ID does not exist"
Sep 16 17:28:55.157387 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:55.157353    2579 scope.go:117] "RemoveContainer" containerID="84faa18e8faf36d1f969d8b0e7cd3ab717e604a6c2ae21561e670c53dfe1613f"
Sep 16 17:28:55.157569 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:55.157552    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84faa18e8faf36d1f969d8b0e7cd3ab717e604a6c2ae21561e670c53dfe1613f"} err="failed to get container status \"84faa18e8faf36d1f969d8b0e7cd3ab717e604a6c2ae21561e670c53dfe1613f\": rpc error: code = NotFound desc = could not find container \"84faa18e8faf36d1f969d8b0e7cd3ab717e604a6c2ae21561e670c53dfe1613f\": container with ID starting with 84faa18e8faf36d1f969d8b0e7cd3ab717e604a6c2ae21561e670c53dfe1613f not found: ID does not exist"
Sep 16 17:28:55.157612 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:55.157571    2579 scope.go:117] "RemoveContainer" containerID="db923488d4f7ebd57a238df7f99586eefd464ab5be7c5284676df6b897c3b734"
Sep 16 17:28:55.157874 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:55.157840    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db923488d4f7ebd57a238df7f99586eefd464ab5be7c5284676df6b897c3b734"} err="failed to get container status \"db923488d4f7ebd57a238df7f99586eefd464ab5be7c5284676df6b897c3b734\": rpc error: code = NotFound desc = could not find container \"db923488d4f7ebd57a238df7f99586eefd464ab5be7c5284676df6b897c3b734\": container with ID starting with db923488d4f7ebd57a238df7f99586eefd464ab5be7c5284676df6b897c3b734 not found: ID does not exist"
Sep 16 17:28:55.157874 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:55.157872    2579 scope.go:117] "RemoveContainer" containerID="7154276c8a3fdc2fc102e0015c16c36591e850d44a73305fcf9d4514c616d25f"
Sep 16 17:28:55.158198 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:55.158173    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7154276c8a3fdc2fc102e0015c16c36591e850d44a73305fcf9d4514c616d25f"} err="failed to get container status \"7154276c8a3fdc2fc102e0015c16c36591e850d44a73305fcf9d4514c616d25f\": rpc error: code = NotFound desc = could not find container \"7154276c8a3fdc2fc102e0015c16c36591e850d44a73305fcf9d4514c616d25f\": container with ID starting with 7154276c8a3fdc2fc102e0015c16c36591e850d44a73305fcf9d4514c616d25f not found: ID does not exist"
Sep 16 17:28:55.158198 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:55.158197    2579 scope.go:117] "RemoveContainer" containerID="6433d2236aad85b04301e28bdad7e4af1109f4cb9ac844bc47af43a367573ba9"
Sep 16 17:28:55.158465 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:55.158441    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6433d2236aad85b04301e28bdad7e4af1109f4cb9ac844bc47af43a367573ba9"} err="failed to get container status \"6433d2236aad85b04301e28bdad7e4af1109f4cb9ac844bc47af43a367573ba9\": rpc error: code = NotFound desc = could not find container \"6433d2236aad85b04301e28bdad7e4af1109f4cb9ac844bc47af43a367573ba9\": container with ID starting with 6433d2236aad85b04301e28bdad7e4af1109f4cb9ac844bc47af43a367573ba9 not found: ID does not exist"
Sep 16 17:28:55.158528 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:55.158469    2579 scope.go:117] "RemoveContainer" containerID="8bc46bc5b55389fcb60ab6a68987c6fdc2ca9ce5cb68aad0b86df70da96c7da1"
Sep 16 17:28:55.158746 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:55.158716    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8bc46bc5b55389fcb60ab6a68987c6fdc2ca9ce5cb68aad0b86df70da96c7da1"} err="failed to get container status \"8bc46bc5b55389fcb60ab6a68987c6fdc2ca9ce5cb68aad0b86df70da96c7da1\": rpc error: code = NotFound desc = could not find container \"8bc46bc5b55389fcb60ab6a68987c6fdc2ca9ce5cb68aad0b86df70da96c7da1\": container with ID starting with 8bc46bc5b55389fcb60ab6a68987c6fdc2ca9ce5cb68aad0b86df70da96c7da1 not found: ID does not exist"
Sep 16 17:28:55.159872 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:55.159840    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 17:28:55.160182 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:55.160169    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2ad2862-5d15-46b2-aaeb-a96ad6af3255" containerName="alertmanager"
Sep 16 17:28:55.160219 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:55.160185    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2ad2862-5d15-46b2-aaeb-a96ad6af3255" containerName="alertmanager"
Sep 16 17:28:55.160219 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:55.160202    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2ad2862-5d15-46b2-aaeb-a96ad6af3255" containerName="kube-rbac-proxy"
Sep 16 17:28:55.160279 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:55.160228    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2ad2862-5d15-46b2-aaeb-a96ad6af3255" containerName="kube-rbac-proxy"
Sep 16 17:28:55.160279 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:55.160239    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2ad2862-5d15-46b2-aaeb-a96ad6af3255" containerName="config-reloader"
Sep 16 17:28:55.160279 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:55.160245    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2ad2862-5d15-46b2-aaeb-a96ad6af3255" containerName="config-reloader"
Sep 16 17:28:55.160279 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:55.160258    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ba8e716-8f2b-4fe4-a140-707f7335fbef" containerName="console"
Sep 16 17:28:55.160279 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:55.160263    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ba8e716-8f2b-4fe4-a140-707f7335fbef" containerName="console"
Sep 16 17:28:55.160279 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:55.160271    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c32493a-a573-4c14-9e85-9f0faef07cd5" containerName="console"
Sep 16 17:28:55.160279 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:55.160279    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c32493a-a573-4c14-9e85-9f0faef07cd5" containerName="console"
Sep 16 17:28:55.160492 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:55.160287    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2ad2862-5d15-46b2-aaeb-a96ad6af3255" containerName="init-config-reloader"
Sep 16 17:28:55.160492 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:55.160292    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2ad2862-5d15-46b2-aaeb-a96ad6af3255" containerName="init-config-reloader"
Sep 16 17:28:55.160492 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:55.160299    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2ad2862-5d15-46b2-aaeb-a96ad6af3255" containerName="kube-rbac-proxy-metric"
Sep 16 17:28:55.160492 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:55.160304    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2ad2862-5d15-46b2-aaeb-a96ad6af3255" containerName="kube-rbac-proxy-metric"
Sep 16 17:28:55.160492 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:55.160310    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2ad2862-5d15-46b2-aaeb-a96ad6af3255" containerName="prom-label-proxy"
Sep 16 17:28:55.160492 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:55.160315    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2ad2862-5d15-46b2-aaeb-a96ad6af3255" containerName="prom-label-proxy"
Sep 16 17:28:55.160492 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:55.160322    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2ad2862-5d15-46b2-aaeb-a96ad6af3255" containerName="kube-rbac-proxy-web"
Sep 16 17:28:55.160492 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:55.160327    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2ad2862-5d15-46b2-aaeb-a96ad6af3255" containerName="kube-rbac-proxy-web"
Sep 16 17:28:55.160492 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:55.160378    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="a2ad2862-5d15-46b2-aaeb-a96ad6af3255" containerName="kube-rbac-proxy-web"
Sep 16 17:28:55.160492 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:55.160386    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="a2ad2862-5d15-46b2-aaeb-a96ad6af3255" containerName="kube-rbac-proxy-metric"
Sep 16 17:28:55.160492 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:55.160393    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="9ba8e716-8f2b-4fe4-a140-707f7335fbef" containerName="console"
Sep 16 17:28:55.160492 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:55.160399    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="3c32493a-a573-4c14-9e85-9f0faef07cd5" containerName="console"
Sep 16 17:28:55.160492 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:55.160404    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="a2ad2862-5d15-46b2-aaeb-a96ad6af3255" containerName="config-reloader"
Sep 16 17:28:55.160492 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:55.160410    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="a2ad2862-5d15-46b2-aaeb-a96ad6af3255" containerName="alertmanager"
Sep 16 17:28:55.160492 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:55.160416    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="a2ad2862-5d15-46b2-aaeb-a96ad6af3255" containerName="kube-rbac-proxy"
Sep 16 17:28:55.160492 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:55.160423    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="a2ad2862-5d15-46b2-aaeb-a96ad6af3255" containerName="prom-label-proxy"
Sep 16 17:28:55.164256 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:55.164241    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 16 17:28:55.166816 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:55.166798    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 16 17:28:55.166922 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:55.166800    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 16 17:28:55.166922 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:55.166802    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 16 17:28:55.167307 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:55.167135    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 16 17:28:55.167307 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:55.167140    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 16 17:28:55.167307 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:55.167190    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 16 17:28:55.167307 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:55.167209    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 16 17:28:55.167307 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:55.167247    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-jkg9r\""
Sep 16 17:28:55.167582 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:55.167565    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 16 17:28:55.172954 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:55.172935    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 16 17:28:55.179734 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:55.179715    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 17:28:55.296573 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:55.296540    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a0504e8b-dfaf-4e78-a021-cecd3ce94cac-config-volume\") pod \"alertmanager-main-0\" (UID: \"a0504e8b-dfaf-4e78-a021-cecd3ce94cac\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 17:28:55.296573 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:55.296577    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a0504e8b-dfaf-4e78-a021-cecd3ce94cac-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"a0504e8b-dfaf-4e78-a021-cecd3ce94cac\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 17:28:55.296759 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:55.296607    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a0504e8b-dfaf-4e78-a021-cecd3ce94cac-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"a0504e8b-dfaf-4e78-a021-cecd3ce94cac\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 17:28:55.296759 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:55.296647    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a0504e8b-dfaf-4e78-a021-cecd3ce94cac-tls-assets\") pod \"alertmanager-main-0\" (UID: \"a0504e8b-dfaf-4e78-a021-cecd3ce94cac\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 17:28:55.296759 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:55.296671    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a0504e8b-dfaf-4e78-a021-cecd3ce94cac-web-config\") pod \"alertmanager-main-0\" (UID: \"a0504e8b-dfaf-4e78-a021-cecd3ce94cac\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 17:28:55.296759 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:55.296740    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a0504e8b-dfaf-4e78-a021-cecd3ce94cac-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"a0504e8b-dfaf-4e78-a021-cecd3ce94cac\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 17:28:55.296913 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:55.296771    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/a0504e8b-dfaf-4e78-a021-cecd3ce94cac-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"a0504e8b-dfaf-4e78-a021-cecd3ce94cac\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 17:28:55.296913 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:55.296823    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a0504e8b-dfaf-4e78-a021-cecd3ce94cac-config-out\") pod \"alertmanager-main-0\" (UID: \"a0504e8b-dfaf-4e78-a021-cecd3ce94cac\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 17:28:55.296913 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:55.296870    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rkpmv\" (UniqueName: \"kubernetes.io/projected/a0504e8b-dfaf-4e78-a021-cecd3ce94cac-kube-api-access-rkpmv\") pod \"alertmanager-main-0\" (UID: \"a0504e8b-dfaf-4e78-a021-cecd3ce94cac\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 17:28:55.296913 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:55.296893    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a0504e8b-dfaf-4e78-a021-cecd3ce94cac-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"a0504e8b-dfaf-4e78-a021-cecd3ce94cac\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 17:28:55.297066 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:55.296933    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a0504e8b-dfaf-4e78-a021-cecd3ce94cac-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"a0504e8b-dfaf-4e78-a021-cecd3ce94cac\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 17:28:55.297066 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:55.296954    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a0504e8b-dfaf-4e78-a021-cecd3ce94cac-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"a0504e8b-dfaf-4e78-a021-cecd3ce94cac\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 17:28:55.297066 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:55.296971    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/a0504e8b-dfaf-4e78-a021-cecd3ce94cac-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"a0504e8b-dfaf-4e78-a021-cecd3ce94cac\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 17:28:55.397595 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:55.397493    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a0504e8b-dfaf-4e78-a021-cecd3ce94cac-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"a0504e8b-dfaf-4e78-a021-cecd3ce94cac\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 17:28:55.397595 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:55.397542    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a0504e8b-dfaf-4e78-a021-cecd3ce94cac-tls-assets\") pod \"alertmanager-main-0\" (UID: \"a0504e8b-dfaf-4e78-a021-cecd3ce94cac\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 17:28:55.397595 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:55.397558    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a0504e8b-dfaf-4e78-a021-cecd3ce94cac-web-config\") pod \"alertmanager-main-0\" (UID: \"a0504e8b-dfaf-4e78-a021-cecd3ce94cac\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 17:28:55.397595 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:55.397584    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a0504e8b-dfaf-4e78-a021-cecd3ce94cac-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"a0504e8b-dfaf-4e78-a021-cecd3ce94cac\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 17:28:55.398245 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:55.397614    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/a0504e8b-dfaf-4e78-a021-cecd3ce94cac-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"a0504e8b-dfaf-4e78-a021-cecd3ce94cac\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 17:28:55.398245 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:55.397760    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a0504e8b-dfaf-4e78-a021-cecd3ce94cac-config-out\") pod \"alertmanager-main-0\" (UID: \"a0504e8b-dfaf-4e78-a021-cecd3ce94cac\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 17:28:55.398245 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:55.397812    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rkpmv\" (UniqueName: \"kubernetes.io/projected/a0504e8b-dfaf-4e78-a021-cecd3ce94cac-kube-api-access-rkpmv\") pod \"alertmanager-main-0\" (UID: \"a0504e8b-dfaf-4e78-a021-cecd3ce94cac\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 17:28:55.398245 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:55.397875    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a0504e8b-dfaf-4e78-a021-cecd3ce94cac-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"a0504e8b-dfaf-4e78-a021-cecd3ce94cac\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 17:28:55.398245 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:55.397940    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a0504e8b-dfaf-4e78-a021-cecd3ce94cac-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"a0504e8b-dfaf-4e78-a021-cecd3ce94cac\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 17:28:55.398245 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:55.397965    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a0504e8b-dfaf-4e78-a021-cecd3ce94cac-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"a0504e8b-dfaf-4e78-a021-cecd3ce94cac\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 17:28:55.398245 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:55.397993    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/a0504e8b-dfaf-4e78-a021-cecd3ce94cac-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"a0504e8b-dfaf-4e78-a021-cecd3ce94cac\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 17:28:55.398245 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:55.398113    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/a0504e8b-dfaf-4e78-a021-cecd3ce94cac-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"a0504e8b-dfaf-4e78-a021-cecd3ce94cac\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 17:28:55.398642 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:55.398409    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a0504e8b-dfaf-4e78-a021-cecd3ce94cac-config-volume\") pod \"alertmanager-main-0\" (UID: \"a0504e8b-dfaf-4e78-a021-cecd3ce94cac\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 17:28:55.398642 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:55.398463    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a0504e8b-dfaf-4e78-a021-cecd3ce94cac-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"a0504e8b-dfaf-4e78-a021-cecd3ce94cac\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 17:28:55.399258 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:55.399125    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a0504e8b-dfaf-4e78-a021-cecd3ce94cac-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"a0504e8b-dfaf-4e78-a021-cecd3ce94cac\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 17:28:55.399542 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:55.399494    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a0504e8b-dfaf-4e78-a021-cecd3ce94cac-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"a0504e8b-dfaf-4e78-a021-cecd3ce94cac\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 17:28:55.400748 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:55.400542    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a0504e8b-dfaf-4e78-a021-cecd3ce94cac-config-out\") pod \"alertmanager-main-0\" (UID: \"a0504e8b-dfaf-4e78-a021-cecd3ce94cac\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 17:28:55.400748 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:55.400699    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a0504e8b-dfaf-4e78-a021-cecd3ce94cac-web-config\") pod \"alertmanager-main-0\" (UID: \"a0504e8b-dfaf-4e78-a021-cecd3ce94cac\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 17:28:55.400930 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:55.400868    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a0504e8b-dfaf-4e78-a021-cecd3ce94cac-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"a0504e8b-dfaf-4e78-a021-cecd3ce94cac\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 17:28:55.400930 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:55.400868    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a0504e8b-dfaf-4e78-a021-cecd3ce94cac-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"a0504e8b-dfaf-4e78-a021-cecd3ce94cac\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 17:28:55.401522 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:55.401481    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a0504e8b-dfaf-4e78-a021-cecd3ce94cac-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"a0504e8b-dfaf-4e78-a021-cecd3ce94cac\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 17:28:55.401685 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:55.401668    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a0504e8b-dfaf-4e78-a021-cecd3ce94cac-tls-assets\") pod \"alertmanager-main-0\" (UID: \"a0504e8b-dfaf-4e78-a021-cecd3ce94cac\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 17:28:55.401912 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:55.401890    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a0504e8b-dfaf-4e78-a021-cecd3ce94cac-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"a0504e8b-dfaf-4e78-a021-cecd3ce94cac\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 17:28:55.401972 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:55.401955    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a0504e8b-dfaf-4e78-a021-cecd3ce94cac-config-volume\") pod \"alertmanager-main-0\" (UID: \"a0504e8b-dfaf-4e78-a021-cecd3ce94cac\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 17:28:55.402593 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:55.402571    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/a0504e8b-dfaf-4e78-a021-cecd3ce94cac-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"a0504e8b-dfaf-4e78-a021-cecd3ce94cac\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 17:28:55.407004 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:55.406987    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkpmv\" (UniqueName: \"kubernetes.io/projected/a0504e8b-dfaf-4e78-a021-cecd3ce94cac-kube-api-access-rkpmv\") pod \"alertmanager-main-0\" (UID: \"a0504e8b-dfaf-4e78-a021-cecd3ce94cac\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 17:28:55.474972 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:55.474934    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 16 17:28:55.760887 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:55.760772    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 17:28:55.763791 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:28:55.763762    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0504e8b_dfaf_4e78_a021_cecd3ce94cac.slice/crio-a7cd6fc488a7aad1bda2d3ea68830692d9623c33fd8db551e6d0ad747291dfdf WatchSource:0}: Error finding container a7cd6fc488a7aad1bda2d3ea68830692d9623c33fd8db551e6d0ad747291dfdf: Status 404 returned error can't find the container with id a7cd6fc488a7aad1bda2d3ea68830692d9623c33fd8db551e6d0ad747291dfdf
Sep 16 17:28:56.098023 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:56.097996    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a2ad2862-5d15-46b2-aaeb-a96ad6af3255" path="/var/lib/kubelet/pods/a2ad2862-5d15-46b2-aaeb-a96ad6af3255/volumes"
Sep 16 17:28:56.105073 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:56.105043    2579 generic.go:356] "Generic (PLEG): container finished" podID="a0504e8b-dfaf-4e78-a021-cecd3ce94cac" containerID="d6af9ee34963a45363a4c7ea637816e90792d88d01057b7548b409dc54917643" exitCode=0
Sep 16 17:28:56.105209 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:56.105091    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a0504e8b-dfaf-4e78-a021-cecd3ce94cac","Type":"ContainerDied","Data":"d6af9ee34963a45363a4c7ea637816e90792d88d01057b7548b409dc54917643"}
Sep 16 17:28:56.105209 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:56.105156    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a0504e8b-dfaf-4e78-a021-cecd3ce94cac","Type":"ContainerStarted","Data":"a7cd6fc488a7aad1bda2d3ea68830692d9623c33fd8db551e6d0ad747291dfdf"}
Sep 16 17:28:57.112073 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:57.112038    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a0504e8b-dfaf-4e78-a021-cecd3ce94cac","Type":"ContainerStarted","Data":"9a0b421350573d7b62326b6e1eb617ee19c8828de5a1450f233690606d702344"}
Sep 16 17:28:57.112444 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:57.112114    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a0504e8b-dfaf-4e78-a021-cecd3ce94cac","Type":"ContainerStarted","Data":"914fb3832613e9531a4ca24b8cc6ccb6edae1e7bf19dfefe63dd2d4dc2606487"}
Sep 16 17:28:57.112444 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:57.112156    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a0504e8b-dfaf-4e78-a021-cecd3ce94cac","Type":"ContainerStarted","Data":"15effb94e1da6ce85ce3ae63aa2db14e64fe3516a70772d0d5cfd181c5cdb512"}
Sep 16 17:28:57.112444 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:57.112197    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a0504e8b-dfaf-4e78-a021-cecd3ce94cac","Type":"ContainerStarted","Data":"07cc2b3f3416fa1f659b0786afa01ce8f71570ec276b28ce49c4e013f63e112b"}
Sep 16 17:28:57.112444 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:57.112233    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a0504e8b-dfaf-4e78-a021-cecd3ce94cac","Type":"ContainerStarted","Data":"a97abedb65c2a1cc2319792628675cccfd4e6406b6201733e603b0c8457ca4f4"}
Sep 16 17:28:57.112444 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:57.112269    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a0504e8b-dfaf-4e78-a021-cecd3ce94cac","Type":"ContainerStarted","Data":"24d2b54966137f5a0441f133377620e0fde4f756f83efaf55c17631c1e3a64b6"}
Sep 16 17:28:57.147473 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:57.147395    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.127668835 podStartE2EDuration="2.147377573s" totalImagesPullingTime="0s" totalInitContainerRuntime="19.708738ms" isStatefulPod=false podCreationTimestamp="2026-09-16 17:28:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:28:57.144806937 +0000 UTC m=+249.614805074" watchObservedRunningTime="2026-09-16 17:28:57.147377573 +0000 UTC m=+249.617375708"
Sep 16 17:28:57.154116 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:57.154088    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-6d7f75957b-q998k"]
Sep 16 17:28:57.156679 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:57.156657    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-6d7f75957b-q998k"
Sep 16 17:28:57.159293 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:57.159270    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-86xqq\""
Sep 16 17:28:57.159825 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:57.159606    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Sep 16 17:28:57.159825 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:57.159637    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Sep 16 17:28:57.159825 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:57.159702    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Sep 16 17:28:57.159825 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:57.159638    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Sep 16 17:28:57.159825 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:57.159611    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Sep 16 17:28:57.164423 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:57.164396    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Sep 16 17:28:57.173837 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:57.173806    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-6d7f75957b-q998k"]
Sep 16 17:28:57.314511 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:57.314457    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9ee7983c-145e-4621-b51c-2083d7e20177-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6d7f75957b-q998k\" (UID: \"9ee7983c-145e-4621-b51c-2083d7e20177\") " pod="openshift-monitoring/telemeter-client-6d7f75957b-q998k"
Sep 16 17:28:57.314511 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:57.314516    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/9ee7983c-145e-4621-b51c-2083d7e20177-secret-telemeter-client\") pod \"telemeter-client-6d7f75957b-q998k\" (UID: \"9ee7983c-145e-4621-b51c-2083d7e20177\") " pod="openshift-monitoring/telemeter-client-6d7f75957b-q998k"
Sep 16 17:28:57.314750 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:57.314640    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/9ee7983c-145e-4621-b51c-2083d7e20177-telemeter-client-tls\") pod \"telemeter-client-6d7f75957b-q998k\" (UID: \"9ee7983c-145e-4621-b51c-2083d7e20177\") " pod="openshift-monitoring/telemeter-client-6d7f75957b-q998k"
Sep 16 17:28:57.314750 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:57.314673    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9ee7983c-145e-4621-b51c-2083d7e20177-serving-certs-ca-bundle\") pod \"telemeter-client-6d7f75957b-q998k\" (UID: \"9ee7983c-145e-4621-b51c-2083d7e20177\") " pod="openshift-monitoring/telemeter-client-6d7f75957b-q998k"
Sep 16 17:28:57.314750 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:57.314705    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wpbvr\" (UniqueName: \"kubernetes.io/projected/9ee7983c-145e-4621-b51c-2083d7e20177-kube-api-access-wpbvr\") pod \"telemeter-client-6d7f75957b-q998k\" (UID: \"9ee7983c-145e-4621-b51c-2083d7e20177\") " pod="openshift-monitoring/telemeter-client-6d7f75957b-q998k"
Sep 16 17:28:57.314750 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:57.314732    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9ee7983c-145e-4621-b51c-2083d7e20177-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6d7f75957b-q998k\" (UID: \"9ee7983c-145e-4621-b51c-2083d7e20177\") " pod="openshift-monitoring/telemeter-client-6d7f75957b-q998k"
Sep 16 17:28:57.314927 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:57.314760    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/9ee7983c-145e-4621-b51c-2083d7e20177-federate-client-tls\") pod \"telemeter-client-6d7f75957b-q998k\" (UID: \"9ee7983c-145e-4621-b51c-2083d7e20177\") " pod="openshift-monitoring/telemeter-client-6d7f75957b-q998k"
Sep 16 17:28:57.314927 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:57.314774    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9ee7983c-145e-4621-b51c-2083d7e20177-metrics-client-ca\") pod \"telemeter-client-6d7f75957b-q998k\" (UID: \"9ee7983c-145e-4621-b51c-2083d7e20177\") " pod="openshift-monitoring/telemeter-client-6d7f75957b-q998k"
Sep 16 17:28:57.415440 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:57.415319    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9ee7983c-145e-4621-b51c-2083d7e20177-serving-certs-ca-bundle\") pod \"telemeter-client-6d7f75957b-q998k\" (UID: \"9ee7983c-145e-4621-b51c-2083d7e20177\") " pod="openshift-monitoring/telemeter-client-6d7f75957b-q998k"
Sep 16 17:28:57.415440 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:57.415376    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wpbvr\" (UniqueName: \"kubernetes.io/projected/9ee7983c-145e-4621-b51c-2083d7e20177-kube-api-access-wpbvr\") pod \"telemeter-client-6d7f75957b-q998k\" (UID: \"9ee7983c-145e-4621-b51c-2083d7e20177\") " pod="openshift-monitoring/telemeter-client-6d7f75957b-q998k"
Sep 16 17:28:57.415440 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:57.415403    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9ee7983c-145e-4621-b51c-2083d7e20177-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6d7f75957b-q998k\" (UID: \"9ee7983c-145e-4621-b51c-2083d7e20177\") " pod="openshift-monitoring/telemeter-client-6d7f75957b-q998k"
Sep 16 17:28:57.415440 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:57.415433    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/9ee7983c-145e-4621-b51c-2083d7e20177-federate-client-tls\") pod \"telemeter-client-6d7f75957b-q998k\" (UID: \"9ee7983c-145e-4621-b51c-2083d7e20177\") " pod="openshift-monitoring/telemeter-client-6d7f75957b-q998k"
Sep 16 17:28:57.415440 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:57.415450    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9ee7983c-145e-4621-b51c-2083d7e20177-metrics-client-ca\") pod \"telemeter-client-6d7f75957b-q998k\" (UID: \"9ee7983c-145e-4621-b51c-2083d7e20177\") " pod="openshift-monitoring/telemeter-client-6d7f75957b-q998k"
Sep 16 17:28:57.415829 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:57.415473    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9ee7983c-145e-4621-b51c-2083d7e20177-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6d7f75957b-q998k\" (UID: \"9ee7983c-145e-4621-b51c-2083d7e20177\") " pod="openshift-monitoring/telemeter-client-6d7f75957b-q998k"
Sep 16 17:28:57.415829 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:57.415499    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/9ee7983c-145e-4621-b51c-2083d7e20177-secret-telemeter-client\") pod \"telemeter-client-6d7f75957b-q998k\" (UID: \"9ee7983c-145e-4621-b51c-2083d7e20177\") " pod="openshift-monitoring/telemeter-client-6d7f75957b-q998k"
Sep 16 17:28:57.415829 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:57.415563    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/9ee7983c-145e-4621-b51c-2083d7e20177-telemeter-client-tls\") pod \"telemeter-client-6d7f75957b-q998k\" (UID: \"9ee7983c-145e-4621-b51c-2083d7e20177\") " pod="openshift-monitoring/telemeter-client-6d7f75957b-q998k"
Sep 16 17:28:57.416183 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:57.416161    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9ee7983c-145e-4621-b51c-2083d7e20177-serving-certs-ca-bundle\") pod \"telemeter-client-6d7f75957b-q998k\" (UID: \"9ee7983c-145e-4621-b51c-2083d7e20177\") " pod="openshift-monitoring/telemeter-client-6d7f75957b-q998k"
Sep 16 17:28:57.416257 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:57.416241    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9ee7983c-145e-4621-b51c-2083d7e20177-metrics-client-ca\") pod \"telemeter-client-6d7f75957b-q998k\" (UID: \"9ee7983c-145e-4621-b51c-2083d7e20177\") " pod="openshift-monitoring/telemeter-client-6d7f75957b-q998k"
Sep 16 17:28:57.416420 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:57.416397    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9ee7983c-145e-4621-b51c-2083d7e20177-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6d7f75957b-q998k\" (UID: \"9ee7983c-145e-4621-b51c-2083d7e20177\") " pod="openshift-monitoring/telemeter-client-6d7f75957b-q998k"
Sep 16 17:28:57.418153 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:57.418126    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/9ee7983c-145e-4621-b51c-2083d7e20177-federate-client-tls\") pod \"telemeter-client-6d7f75957b-q998k\" (UID: \"9ee7983c-145e-4621-b51c-2083d7e20177\") " pod="openshift-monitoring/telemeter-client-6d7f75957b-q998k"
Sep 16 17:28:57.418251 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:57.418153    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/9ee7983c-145e-4621-b51c-2083d7e20177-secret-telemeter-client\") pod \"telemeter-client-6d7f75957b-q998k\" (UID: \"9ee7983c-145e-4621-b51c-2083d7e20177\") " pod="openshift-monitoring/telemeter-client-6d7f75957b-q998k"
Sep 16 17:28:57.418414 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:57.418392    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9ee7983c-145e-4621-b51c-2083d7e20177-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6d7f75957b-q998k\" (UID: \"9ee7983c-145e-4621-b51c-2083d7e20177\") " pod="openshift-monitoring/telemeter-client-6d7f75957b-q998k"
Sep 16 17:28:57.418566 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:57.418552    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/9ee7983c-145e-4621-b51c-2083d7e20177-telemeter-client-tls\") pod \"telemeter-client-6d7f75957b-q998k\" (UID: \"9ee7983c-145e-4621-b51c-2083d7e20177\") " pod="openshift-monitoring/telemeter-client-6d7f75957b-q998k"
Sep 16 17:28:57.429081 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:57.429056    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wpbvr\" (UniqueName: \"kubernetes.io/projected/9ee7983c-145e-4621-b51c-2083d7e20177-kube-api-access-wpbvr\") pod \"telemeter-client-6d7f75957b-q998k\" (UID: \"9ee7983c-145e-4621-b51c-2083d7e20177\") " pod="openshift-monitoring/telemeter-client-6d7f75957b-q998k"
Sep 16 17:28:57.473546 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:57.473517    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-6d7f75957b-q998k"
Sep 16 17:28:57.759391 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:57.759361    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-6d7f75957b-q998k"]
Sep 16 17:28:57.760348 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:28:57.760325    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ee7983c_145e_4621_b51c_2083d7e20177.slice/crio-5f7c59bd2f28b5b542731cc9e16737766e96b58e0d556a2901ba525429315442 WatchSource:0}: Error finding container 5f7c59bd2f28b5b542731cc9e16737766e96b58e0d556a2901ba525429315442: Status 404 returned error can't find the container with id 5f7c59bd2f28b5b542731cc9e16737766e96b58e0d556a2901ba525429315442
Sep 16 17:28:58.117311 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:58.117277    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6d7f75957b-q998k" event={"ID":"9ee7983c-145e-4621-b51c-2083d7e20177","Type":"ContainerStarted","Data":"5f7c59bd2f28b5b542731cc9e16737766e96b58e0d556a2901ba525429315442"}
Sep 16 17:28:59.934509 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:59.934420    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bc719829-66b1-4a84-b994-e71cf111dafa-metrics-certs\") pod \"network-metrics-daemon-xfgjq\" (UID: \"bc719829-66b1-4a84-b994-e71cf111dafa\") " pod="openshift-multus/network-metrics-daemon-xfgjq"
Sep 16 17:28:59.936825 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:59.936800    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bc719829-66b1-4a84-b994-e71cf111dafa-metrics-certs\") pod \"network-metrics-daemon-xfgjq\" (UID: \"bc719829-66b1-4a84-b994-e71cf111dafa\") " pod="openshift-multus/network-metrics-daemon-xfgjq"
Sep 16 17:28:59.999584 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:28:59.999551    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-q452j\""
Sep 16 17:29:00.007859 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:29:00.007814    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xfgjq"
Sep 16 17:29:00.127542 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:29:00.127506    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6d7f75957b-q998k" event={"ID":"9ee7983c-145e-4621-b51c-2083d7e20177","Type":"ContainerStarted","Data":"16ba3a56de58e331be3c05b0b9e4dc4218c747308524b6b4d91ad0f02fcf17fe"}
Sep 16 17:29:00.127700 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:29:00.127568    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6d7f75957b-q998k" event={"ID":"9ee7983c-145e-4621-b51c-2083d7e20177","Type":"ContainerStarted","Data":"bfe93e475b3245314ac14acb7310c950c92530376a2d6aa608df89541278e435"}
Sep 16 17:29:00.127700 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:29:00.127587    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6d7f75957b-q998k" event={"ID":"9ee7983c-145e-4621-b51c-2083d7e20177","Type":"ContainerStarted","Data":"13aec4ccd19c48bf23fc66ada73c4e9ad0cbeba9a4caf1de2a60d630f27e4bee"}
Sep 16 17:29:00.158740 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:29:00.158689    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-6d7f75957b-q998k" podStartSLOduration=1.296925931 podStartE2EDuration="3.158673118s" totalImagesPullingTime="1.861747187s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:28:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:29:00.158140712 +0000 UTC m=+252.628138883" watchObservedRunningTime="2026-09-16 17:29:00.158673118 +0000 UTC m=+252.628671252"
Sep 16 17:29:00.295802 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:29:00.295774    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-xfgjq"]
Sep 16 17:29:00.298887 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:29:00.298836    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc719829_66b1_4a84_b994_e71cf111dafa.slice/crio-cf4c67eef2812a0707b90e7cbecb62937019929501fac513316bd8a0967b3d8b WatchSource:0}: Error finding container cf4c67eef2812a0707b90e7cbecb62937019929501fac513316bd8a0967b3d8b: Status 404 returned error can't find the container with id cf4c67eef2812a0707b90e7cbecb62937019929501fac513316bd8a0967b3d8b
Sep 16 17:29:00.300074 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:29:00.300051    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5f7b9bb969-gdxqv"]
Sep 16 17:29:00.302912 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:29:00.302891    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5f7b9bb969-gdxqv"
Sep 16 17:29:00.322644 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:29:00.322617    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5f7b9bb969-gdxqv"]
Sep 16 17:29:00.439432 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:29:00.439392    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4020fe82-9799-424d-a7c0-9b9cad1eab40-console-config\") pod \"console-5f7b9bb969-gdxqv\" (UID: \"4020fe82-9799-424d-a7c0-9b9cad1eab40\") " pod="openshift-console/console-5f7b9bb969-gdxqv"
Sep 16 17:29:00.439432 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:29:00.439436    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4020fe82-9799-424d-a7c0-9b9cad1eab40-console-oauth-config\") pod \"console-5f7b9bb969-gdxqv\" (UID: \"4020fe82-9799-424d-a7c0-9b9cad1eab40\") " pod="openshift-console/console-5f7b9bb969-gdxqv"
Sep 16 17:29:00.439639 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:29:00.439456    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4020fe82-9799-424d-a7c0-9b9cad1eab40-service-ca\") pod \"console-5f7b9bb969-gdxqv\" (UID: \"4020fe82-9799-424d-a7c0-9b9cad1eab40\") " pod="openshift-console/console-5f7b9bb969-gdxqv"
Sep 16 17:29:00.439639 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:29:00.439490    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4020fe82-9799-424d-a7c0-9b9cad1eab40-oauth-serving-cert\") pod \"console-5f7b9bb969-gdxqv\" (UID: \"4020fe82-9799-424d-a7c0-9b9cad1eab40\") " pod="openshift-console/console-5f7b9bb969-gdxqv"
Sep 16 17:29:00.439639 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:29:00.439522    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4020fe82-9799-424d-a7c0-9b9cad1eab40-console-serving-cert\") pod \"console-5f7b9bb969-gdxqv\" (UID: \"4020fe82-9799-424d-a7c0-9b9cad1eab40\") " pod="openshift-console/console-5f7b9bb969-gdxqv"
Sep 16 17:29:00.439731 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:29:00.439634    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4020fe82-9799-424d-a7c0-9b9cad1eab40-trusted-ca-bundle\") pod \"console-5f7b9bb969-gdxqv\" (UID: \"4020fe82-9799-424d-a7c0-9b9cad1eab40\") " pod="openshift-console/console-5f7b9bb969-gdxqv"
Sep 16 17:29:00.439731 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:29:00.439667    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b64nr\" (UniqueName: \"kubernetes.io/projected/4020fe82-9799-424d-a7c0-9b9cad1eab40-kube-api-access-b64nr\") pod \"console-5f7b9bb969-gdxqv\" (UID: \"4020fe82-9799-424d-a7c0-9b9cad1eab40\") " pod="openshift-console/console-5f7b9bb969-gdxqv"
Sep 16 17:29:00.540113 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:29:00.540080    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4020fe82-9799-424d-a7c0-9b9cad1eab40-trusted-ca-bundle\") pod \"console-5f7b9bb969-gdxqv\" (UID: \"4020fe82-9799-424d-a7c0-9b9cad1eab40\") " pod="openshift-console/console-5f7b9bb969-gdxqv"
Sep 16 17:29:00.540221 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:29:00.540117    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b64nr\" (UniqueName: \"kubernetes.io/projected/4020fe82-9799-424d-a7c0-9b9cad1eab40-kube-api-access-b64nr\") pod \"console-5f7b9bb969-gdxqv\" (UID: \"4020fe82-9799-424d-a7c0-9b9cad1eab40\") " pod="openshift-console/console-5f7b9bb969-gdxqv"
Sep 16 17:29:00.540221 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:29:00.540165    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4020fe82-9799-424d-a7c0-9b9cad1eab40-console-config\") pod \"console-5f7b9bb969-gdxqv\" (UID: \"4020fe82-9799-424d-a7c0-9b9cad1eab40\") " pod="openshift-console/console-5f7b9bb969-gdxqv"
Sep 16 17:29:00.540221 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:29:00.540187    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4020fe82-9799-424d-a7c0-9b9cad1eab40-console-oauth-config\") pod \"console-5f7b9bb969-gdxqv\" (UID: \"4020fe82-9799-424d-a7c0-9b9cad1eab40\") " pod="openshift-console/console-5f7b9bb969-gdxqv"
Sep 16 17:29:00.540221 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:29:00.540204    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4020fe82-9799-424d-a7c0-9b9cad1eab40-service-ca\") pod \"console-5f7b9bb969-gdxqv\" (UID: \"4020fe82-9799-424d-a7c0-9b9cad1eab40\") " pod="openshift-console/console-5f7b9bb969-gdxqv"
Sep 16 17:29:00.540405 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:29:00.540236    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4020fe82-9799-424d-a7c0-9b9cad1eab40-oauth-serving-cert\") pod \"console-5f7b9bb969-gdxqv\" (UID: \"4020fe82-9799-424d-a7c0-9b9cad1eab40\") " pod="openshift-console/console-5f7b9bb969-gdxqv"
Sep 16 17:29:00.540405 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:29:00.540262    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4020fe82-9799-424d-a7c0-9b9cad1eab40-console-serving-cert\") pod \"console-5f7b9bb969-gdxqv\" (UID: \"4020fe82-9799-424d-a7c0-9b9cad1eab40\") " pod="openshift-console/console-5f7b9bb969-gdxqv"
Sep 16 17:29:00.541033 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:29:00.540999    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4020fe82-9799-424d-a7c0-9b9cad1eab40-service-ca\") pod \"console-5f7b9bb969-gdxqv\" (UID: \"4020fe82-9799-424d-a7c0-9b9cad1eab40\") " pod="openshift-console/console-5f7b9bb969-gdxqv"
Sep 16 17:29:00.541033 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:29:00.541023    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4020fe82-9799-424d-a7c0-9b9cad1eab40-oauth-serving-cert\") pod \"console-5f7b9bb969-gdxqv\" (UID: \"4020fe82-9799-424d-a7c0-9b9cad1eab40\") " pod="openshift-console/console-5f7b9bb969-gdxqv"
Sep 16 17:29:00.541241 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:29:00.541078    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4020fe82-9799-424d-a7c0-9b9cad1eab40-console-config\") pod \"console-5f7b9bb969-gdxqv\" (UID: \"4020fe82-9799-424d-a7c0-9b9cad1eab40\") " pod="openshift-console/console-5f7b9bb969-gdxqv"
Sep 16 17:29:00.541241 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:29:00.541087    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4020fe82-9799-424d-a7c0-9b9cad1eab40-trusted-ca-bundle\") pod \"console-5f7b9bb969-gdxqv\" (UID: \"4020fe82-9799-424d-a7c0-9b9cad1eab40\") " pod="openshift-console/console-5f7b9bb969-gdxqv"
Sep 16 17:29:00.542856 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:29:00.542820    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4020fe82-9799-424d-a7c0-9b9cad1eab40-console-oauth-config\") pod \"console-5f7b9bb969-gdxqv\" (UID: \"4020fe82-9799-424d-a7c0-9b9cad1eab40\") " pod="openshift-console/console-5f7b9bb969-gdxqv"
Sep 16 17:29:00.542933 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:29:00.542911    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4020fe82-9799-424d-a7c0-9b9cad1eab40-console-serving-cert\") pod \"console-5f7b9bb969-gdxqv\" (UID: \"4020fe82-9799-424d-a7c0-9b9cad1eab40\") " pod="openshift-console/console-5f7b9bb969-gdxqv"
Sep 16 17:29:00.549870 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:29:00.549815    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b64nr\" (UniqueName: \"kubernetes.io/projected/4020fe82-9799-424d-a7c0-9b9cad1eab40-kube-api-access-b64nr\") pod \"console-5f7b9bb969-gdxqv\" (UID: \"4020fe82-9799-424d-a7c0-9b9cad1eab40\") " pod="openshift-console/console-5f7b9bb969-gdxqv"
Sep 16 17:29:00.612746 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:29:00.612717    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5f7b9bb969-gdxqv"
Sep 16 17:29:00.899951 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:29:00.899869    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5f7b9bb969-gdxqv"]
Sep 16 17:29:01.134505 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:29:01.134465    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xfgjq" event={"ID":"bc719829-66b1-4a84-b994-e71cf111dafa","Type":"ContainerStarted","Data":"cf4c67eef2812a0707b90e7cbecb62937019929501fac513316bd8a0967b3d8b"}
Sep 16 17:29:01.136408 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:29:01.136374    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5f7b9bb969-gdxqv" event={"ID":"4020fe82-9799-424d-a7c0-9b9cad1eab40","Type":"ContainerStarted","Data":"a62834b2ee9746f21f8eb26a604d38df6e3f932ef66a4bafa3972532d0a2f4a6"}
Sep 16 17:29:01.136564 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:29:01.136429    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5f7b9bb969-gdxqv" event={"ID":"4020fe82-9799-424d-a7c0-9b9cad1eab40","Type":"ContainerStarted","Data":"7b0a4f580574dc0eac2201edd422fe789da7932d23e3bc6d3fda9e92a0cc547b"}
Sep 16 17:29:02.141558 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:29:02.141520    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xfgjq" event={"ID":"bc719829-66b1-4a84-b994-e71cf111dafa","Type":"ContainerStarted","Data":"ef3d56de7ecf79342aea721f3432d407934cc450743084dfdff7fbfadbc0cd18"}
Sep 16 17:29:02.142062 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:29:02.141588    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xfgjq" event={"ID":"bc719829-66b1-4a84-b994-e71cf111dafa","Type":"ContainerStarted","Data":"bb78d84929cb57bb2dcc2b147d2e855b32dfc6ca0f4832578326ef6aaad6108d"}
Sep 16 17:29:02.158438 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:29:02.158392    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-xfgjq" podStartSLOduration=253.179809111 podStartE2EDuration="4m14.158376329s" totalImagesPullingTime="978.567218ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:24:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:29:02.157067685 +0000 UTC m=+254.627065821" watchObservedRunningTime="2026-09-16 17:29:02.158376329 +0000 UTC m=+254.628374463"
Sep 16 17:29:02.158580 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:29:02.158518    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5f7b9bb969-gdxqv" podStartSLOduration=2.158513023 podStartE2EDuration="2.158513023s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:29:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:29:01.164135415 +0000 UTC m=+253.634133551" watchObservedRunningTime="2026-09-16 17:29:02.158513023 +0000 UTC m=+254.628511158"
Sep 16 17:29:10.613933 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:29:10.613821    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5f7b9bb969-gdxqv"
Sep 16 17:29:10.613933 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:29:10.613909    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5f7b9bb969-gdxqv"
Sep 16 17:29:10.618476 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:29:10.618454    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5f7b9bb969-gdxqv"
Sep 16 17:29:11.186317 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:29:11.186287    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5f7b9bb969-gdxqv"
Sep 16 17:29:11.243912 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:29:11.243880    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-97c9d775b-jcdpd"]
Sep 16 17:29:36.264857 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:29:36.264801    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-97c9d775b-jcdpd" podUID="817c0617-8912-4e88-850e-f8b89ac8b191" containerName="console" containerID="cri-o://a5041b5023c85ddcf8e562e0dd26226b460a49f00e2003f9ae4d0a3a650f0c6a" gracePeriod=15
Sep 16 17:29:36.508107 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:29:36.508082    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-97c9d775b-jcdpd"
Sep 16 17:29:36.644514 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:29:36.644480    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/817c0617-8912-4e88-850e-f8b89ac8b191-console-config\") pod \"817c0617-8912-4e88-850e-f8b89ac8b191\" (UID: \"817c0617-8912-4e88-850e-f8b89ac8b191\") "
Sep 16 17:29:36.644715 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:29:36.644531    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/817c0617-8912-4e88-850e-f8b89ac8b191-trusted-ca-bundle\") pod \"817c0617-8912-4e88-850e-f8b89ac8b191\" (UID: \"817c0617-8912-4e88-850e-f8b89ac8b191\") "
Sep 16 17:29:36.644715 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:29:36.644557    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/817c0617-8912-4e88-850e-f8b89ac8b191-oauth-serving-cert\") pod \"817c0617-8912-4e88-850e-f8b89ac8b191\" (UID: \"817c0617-8912-4e88-850e-f8b89ac8b191\") "
Sep 16 17:29:36.644715 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:29:36.644580    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/817c0617-8912-4e88-850e-f8b89ac8b191-service-ca\") pod \"817c0617-8912-4e88-850e-f8b89ac8b191\" (UID: \"817c0617-8912-4e88-850e-f8b89ac8b191\") "
Sep 16 17:29:36.644715 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:29:36.644634    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cwt4v\" (UniqueName: \"kubernetes.io/projected/817c0617-8912-4e88-850e-f8b89ac8b191-kube-api-access-cwt4v\") pod \"817c0617-8912-4e88-850e-f8b89ac8b191\" (UID: \"817c0617-8912-4e88-850e-f8b89ac8b191\") "
Sep 16 17:29:36.644715 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:29:36.644685    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/817c0617-8912-4e88-850e-f8b89ac8b191-console-oauth-config\") pod \"817c0617-8912-4e88-850e-f8b89ac8b191\" (UID: \"817c0617-8912-4e88-850e-f8b89ac8b191\") "
Sep 16 17:29:36.644715 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:29:36.644711    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/817c0617-8912-4e88-850e-f8b89ac8b191-console-serving-cert\") pod \"817c0617-8912-4e88-850e-f8b89ac8b191\" (UID: \"817c0617-8912-4e88-850e-f8b89ac8b191\") "
Sep 16 17:29:36.645031 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:29:36.644953    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/817c0617-8912-4e88-850e-f8b89ac8b191-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "817c0617-8912-4e88-850e-f8b89ac8b191" (UID: "817c0617-8912-4e88-850e-f8b89ac8b191"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 17:29:36.645031 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:29:36.644995    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/817c0617-8912-4e88-850e-f8b89ac8b191-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "817c0617-8912-4e88-850e-f8b89ac8b191" (UID: "817c0617-8912-4e88-850e-f8b89ac8b191"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 17:29:36.645376 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:29:36.645355    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/817c0617-8912-4e88-850e-f8b89ac8b191-service-ca" (OuterVolumeSpecName: "service-ca") pod "817c0617-8912-4e88-850e-f8b89ac8b191" (UID: "817c0617-8912-4e88-850e-f8b89ac8b191"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 17:29:36.645549 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:29:36.645523    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/817c0617-8912-4e88-850e-f8b89ac8b191-console-config" (OuterVolumeSpecName: "console-config") pod "817c0617-8912-4e88-850e-f8b89ac8b191" (UID: "817c0617-8912-4e88-850e-f8b89ac8b191"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 17:29:36.647100 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:29:36.647076    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/817c0617-8912-4e88-850e-f8b89ac8b191-kube-api-access-cwt4v" (OuterVolumeSpecName: "kube-api-access-cwt4v") pod "817c0617-8912-4e88-850e-f8b89ac8b191" (UID: "817c0617-8912-4e88-850e-f8b89ac8b191"). InnerVolumeSpecName "kube-api-access-cwt4v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 17:29:36.647348 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:29:36.647327    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/817c0617-8912-4e88-850e-f8b89ac8b191-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "817c0617-8912-4e88-850e-f8b89ac8b191" (UID: "817c0617-8912-4e88-850e-f8b89ac8b191"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 17:29:36.647441 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:29:36.647356    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/817c0617-8912-4e88-850e-f8b89ac8b191-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "817c0617-8912-4e88-850e-f8b89ac8b191" (UID: "817c0617-8912-4e88-850e-f8b89ac8b191"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 17:29:36.745758 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:29:36.745708    2579 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/817c0617-8912-4e88-850e-f8b89ac8b191-console-oauth-config\") on node \"ip-10-0-138-73.ec2.internal\" DevicePath \"\""
Sep 16 17:29:36.745758 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:29:36.745751    2579 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/817c0617-8912-4e88-850e-f8b89ac8b191-console-serving-cert\") on node \"ip-10-0-138-73.ec2.internal\" DevicePath \"\""
Sep 16 17:29:36.745758 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:29:36.745760    2579 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/817c0617-8912-4e88-850e-f8b89ac8b191-console-config\") on node \"ip-10-0-138-73.ec2.internal\" DevicePath \"\""
Sep 16 17:29:36.745758 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:29:36.745771    2579 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/817c0617-8912-4e88-850e-f8b89ac8b191-trusted-ca-bundle\") on node \"ip-10-0-138-73.ec2.internal\" DevicePath \"\""
Sep 16 17:29:36.745758 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:29:36.745779    2579 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/817c0617-8912-4e88-850e-f8b89ac8b191-oauth-serving-cert\") on node \"ip-10-0-138-73.ec2.internal\" DevicePath \"\""
Sep 16 17:29:36.746078 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:29:36.745806    2579 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/817c0617-8912-4e88-850e-f8b89ac8b191-service-ca\") on node \"ip-10-0-138-73.ec2.internal\" DevicePath \"\""
Sep 16 17:29:36.746078 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:29:36.745816    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cwt4v\" (UniqueName: \"kubernetes.io/projected/817c0617-8912-4e88-850e-f8b89ac8b191-kube-api-access-cwt4v\") on node \"ip-10-0-138-73.ec2.internal\" DevicePath \"\""
Sep 16 17:29:37.293985 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:29:37.293952    2579 generic.go:356] "Generic (PLEG): container finished" podID="817c0617-8912-4e88-850e-f8b89ac8b191" containerID="a5041b5023c85ddcf8e562e0dd26226b460a49f00e2003f9ae4d0a3a650f0c6a" exitCode=0
Sep 16 17:29:37.294472 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:29:37.294019    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-97c9d775b-jcdpd"
Sep 16 17:29:37.294472 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:29:37.294031    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-97c9d775b-jcdpd" event={"ID":"817c0617-8912-4e88-850e-f8b89ac8b191","Type":"ContainerDied","Data":"a5041b5023c85ddcf8e562e0dd26226b460a49f00e2003f9ae4d0a3a650f0c6a"}
Sep 16 17:29:37.294472 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:29:37.294093    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-97c9d775b-jcdpd" event={"ID":"817c0617-8912-4e88-850e-f8b89ac8b191","Type":"ContainerDied","Data":"46f7accf1e49ad536e3b00b48aeb5018d47f1a45abcd9f5f8ff5ffc2edc68650"}
Sep 16 17:29:37.294472 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:29:37.294112    2579 scope.go:117] "RemoveContainer" containerID="a5041b5023c85ddcf8e562e0dd26226b460a49f00e2003f9ae4d0a3a650f0c6a"
Sep 16 17:29:37.303313 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:29:37.303294    2579 scope.go:117] "RemoveContainer" containerID="a5041b5023c85ddcf8e562e0dd26226b460a49f00e2003f9ae4d0a3a650f0c6a"
Sep 16 17:29:37.303585 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:29:37.303565    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5041b5023c85ddcf8e562e0dd26226b460a49f00e2003f9ae4d0a3a650f0c6a\": container with ID starting with a5041b5023c85ddcf8e562e0dd26226b460a49f00e2003f9ae4d0a3a650f0c6a not found: ID does not exist" containerID="a5041b5023c85ddcf8e562e0dd26226b460a49f00e2003f9ae4d0a3a650f0c6a"
Sep 16 17:29:37.303630 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:29:37.303594    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5041b5023c85ddcf8e562e0dd26226b460a49f00e2003f9ae4d0a3a650f0c6a"} err="failed to get container status \"a5041b5023c85ddcf8e562e0dd26226b460a49f00e2003f9ae4d0a3a650f0c6a\": rpc error: code = NotFound desc = could not find container \"a5041b5023c85ddcf8e562e0dd26226b460a49f00e2003f9ae4d0a3a650f0c6a\": container with ID starting with a5041b5023c85ddcf8e562e0dd26226b460a49f00e2003f9ae4d0a3a650f0c6a not found: ID does not exist"
Sep 16 17:29:37.316397 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:29:37.316373    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-97c9d775b-jcdpd"]
Sep 16 17:29:37.321975 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:29:37.321951    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-97c9d775b-jcdpd"]
Sep 16 17:29:38.099004 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:29:38.098961    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="817c0617-8912-4e88-850e-f8b89ac8b191" path="/var/lib/kubelet/pods/817c0617-8912-4e88-850e-f8b89ac8b191/volumes"
Sep 16 17:29:47.950205 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:29:47.950181    2579 kubelet.go:1648] "Image garbage collection succeeded"
Sep 16 17:30:14.823457 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:30:14.823427    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-794c96db68-55xsz"]
Sep 16 17:30:14.825926 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:30:14.823763    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="817c0617-8912-4e88-850e-f8b89ac8b191" containerName="console"
Sep 16 17:30:14.825926 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:30:14.823774    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="817c0617-8912-4e88-850e-f8b89ac8b191" containerName="console"
Sep 16 17:30:14.825926 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:30:14.823863    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="817c0617-8912-4e88-850e-f8b89ac8b191" containerName="console"
Sep 16 17:30:14.826641 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:30:14.826625    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-794c96db68-55xsz"
Sep 16 17:30:14.848582 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:30:14.848554    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-794c96db68-55xsz"]
Sep 16 17:30:14.853322 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:30:14.853297    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/528737a5-a963-4703-bc0e-edd77943a7e7-service-ca\") pod \"console-794c96db68-55xsz\" (UID: \"528737a5-a963-4703-bc0e-edd77943a7e7\") " pod="openshift-console/console-794c96db68-55xsz"
Sep 16 17:30:14.853424 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:30:14.853400    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/528737a5-a963-4703-bc0e-edd77943a7e7-console-serving-cert\") pod \"console-794c96db68-55xsz\" (UID: \"528737a5-a963-4703-bc0e-edd77943a7e7\") " pod="openshift-console/console-794c96db68-55xsz"
Sep 16 17:30:14.853463 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:30:14.853437    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjzf4\" (UniqueName: \"kubernetes.io/projected/528737a5-a963-4703-bc0e-edd77943a7e7-kube-api-access-tjzf4\") pod \"console-794c96db68-55xsz\" (UID: \"528737a5-a963-4703-bc0e-edd77943a7e7\") " pod="openshift-console/console-794c96db68-55xsz"
Sep 16 17:30:14.853463 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:30:14.853457    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/528737a5-a963-4703-bc0e-edd77943a7e7-console-oauth-config\") pod \"console-794c96db68-55xsz\" (UID: \"528737a5-a963-4703-bc0e-edd77943a7e7\") " pod="openshift-console/console-794c96db68-55xsz"
Sep 16 17:30:14.853538 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:30:14.853481    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/528737a5-a963-4703-bc0e-edd77943a7e7-oauth-serving-cert\") pod \"console-794c96db68-55xsz\" (UID: \"528737a5-a963-4703-bc0e-edd77943a7e7\") " pod="openshift-console/console-794c96db68-55xsz"
Sep 16 17:30:14.853538 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:30:14.853526    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/528737a5-a963-4703-bc0e-edd77943a7e7-trusted-ca-bundle\") pod \"console-794c96db68-55xsz\" (UID: \"528737a5-a963-4703-bc0e-edd77943a7e7\") " pod="openshift-console/console-794c96db68-55xsz"
Sep 16 17:30:14.853608 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:30:14.853555    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/528737a5-a963-4703-bc0e-edd77943a7e7-console-config\") pod \"console-794c96db68-55xsz\" (UID: \"528737a5-a963-4703-bc0e-edd77943a7e7\") " pod="openshift-console/console-794c96db68-55xsz"
Sep 16 17:30:14.954343 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:30:14.954300    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjzf4\" (UniqueName: \"kubernetes.io/projected/528737a5-a963-4703-bc0e-edd77943a7e7-kube-api-access-tjzf4\") pod \"console-794c96db68-55xsz\" (UID: \"528737a5-a963-4703-bc0e-edd77943a7e7\") " pod="openshift-console/console-794c96db68-55xsz"
Sep 16 17:30:14.954343 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:30:14.954345    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/528737a5-a963-4703-bc0e-edd77943a7e7-console-oauth-config\") pod \"console-794c96db68-55xsz\" (UID: \"528737a5-a963-4703-bc0e-edd77943a7e7\") " pod="openshift-console/console-794c96db68-55xsz"
Sep 16 17:30:14.954601 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:30:14.954371    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/528737a5-a963-4703-bc0e-edd77943a7e7-oauth-serving-cert\") pod \"console-794c96db68-55xsz\" (UID: \"528737a5-a963-4703-bc0e-edd77943a7e7\") " pod="openshift-console/console-794c96db68-55xsz"
Sep 16 17:30:14.954601 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:30:14.954407    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/528737a5-a963-4703-bc0e-edd77943a7e7-trusted-ca-bundle\") pod \"console-794c96db68-55xsz\" (UID: \"528737a5-a963-4703-bc0e-edd77943a7e7\") " pod="openshift-console/console-794c96db68-55xsz"
Sep 16 17:30:14.954601 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:30:14.954436    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/528737a5-a963-4703-bc0e-edd77943a7e7-console-config\") pod \"console-794c96db68-55xsz\" (UID: \"528737a5-a963-4703-bc0e-edd77943a7e7\") " pod="openshift-console/console-794c96db68-55xsz"
Sep 16 17:30:14.954601 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:30:14.954479    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/528737a5-a963-4703-bc0e-edd77943a7e7-service-ca\") pod \"console-794c96db68-55xsz\" (UID: \"528737a5-a963-4703-bc0e-edd77943a7e7\") " pod="openshift-console/console-794c96db68-55xsz"
Sep 16 17:30:14.954601 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:30:14.954544    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/528737a5-a963-4703-bc0e-edd77943a7e7-console-serving-cert\") pod \"console-794c96db68-55xsz\" (UID: \"528737a5-a963-4703-bc0e-edd77943a7e7\") " pod="openshift-console/console-794c96db68-55xsz"
Sep 16 17:30:14.955336 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:30:14.955305    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/528737a5-a963-4703-bc0e-edd77943a7e7-oauth-serving-cert\") pod \"console-794c96db68-55xsz\" (UID: \"528737a5-a963-4703-bc0e-edd77943a7e7\") " pod="openshift-console/console-794c96db68-55xsz"
Sep 16 17:30:14.955478 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:30:14.955305    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/528737a5-a963-4703-bc0e-edd77943a7e7-service-ca\") pod \"console-794c96db68-55xsz\" (UID: \"528737a5-a963-4703-bc0e-edd77943a7e7\") " pod="openshift-console/console-794c96db68-55xsz"
Sep 16 17:30:14.955478 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:30:14.955367    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/528737a5-a963-4703-bc0e-edd77943a7e7-console-config\") pod \"console-794c96db68-55xsz\" (UID: \"528737a5-a963-4703-bc0e-edd77943a7e7\") " pod="openshift-console/console-794c96db68-55xsz"
Sep 16 17:30:14.955478 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:30:14.955446    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/528737a5-a963-4703-bc0e-edd77943a7e7-trusted-ca-bundle\") pod \"console-794c96db68-55xsz\" (UID: \"528737a5-a963-4703-bc0e-edd77943a7e7\") " pod="openshift-console/console-794c96db68-55xsz"
Sep 16 17:30:14.956787 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:30:14.956759    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/528737a5-a963-4703-bc0e-edd77943a7e7-console-oauth-config\") pod \"console-794c96db68-55xsz\" (UID: \"528737a5-a963-4703-bc0e-edd77943a7e7\") " pod="openshift-console/console-794c96db68-55xsz"
Sep 16 17:30:14.957043 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:30:14.957017    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/528737a5-a963-4703-bc0e-edd77943a7e7-console-serving-cert\") pod \"console-794c96db68-55xsz\" (UID: \"528737a5-a963-4703-bc0e-edd77943a7e7\") " pod="openshift-console/console-794c96db68-55xsz"
Sep 16 17:30:14.962768 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:30:14.962747    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjzf4\" (UniqueName: \"kubernetes.io/projected/528737a5-a963-4703-bc0e-edd77943a7e7-kube-api-access-tjzf4\") pod \"console-794c96db68-55xsz\" (UID: \"528737a5-a963-4703-bc0e-edd77943a7e7\") " pod="openshift-console/console-794c96db68-55xsz"
Sep 16 17:30:15.136393 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:30:15.136309    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-794c96db68-55xsz"
Sep 16 17:30:15.413365 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:30:15.413338    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-794c96db68-55xsz"]
Sep 16 17:30:15.416687 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:30:15.416646    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod528737a5_a963_4703_bc0e_edd77943a7e7.slice/crio-2ae9e78b15bc627ae02da011011d6a2a754f737b4b322039aefbfdf0aba1110f WatchSource:0}: Error finding container 2ae9e78b15bc627ae02da011011d6a2a754f737b4b322039aefbfdf0aba1110f: Status 404 returned error can't find the container with id 2ae9e78b15bc627ae02da011011d6a2a754f737b4b322039aefbfdf0aba1110f
Sep 16 17:30:15.449602 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:30:15.449573    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-794c96db68-55xsz" event={"ID":"528737a5-a963-4703-bc0e-edd77943a7e7","Type":"ContainerStarted","Data":"2ae9e78b15bc627ae02da011011d6a2a754f737b4b322039aefbfdf0aba1110f"}
Sep 16 17:30:16.455801 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:30:16.455765    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-794c96db68-55xsz" event={"ID":"528737a5-a963-4703-bc0e-edd77943a7e7","Type":"ContainerStarted","Data":"8eadd0f429e5ac5babc6b07d2f51b1e16bf7271328a32b13cd32c6f597e9a142"}
Sep 16 17:30:16.474091 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:30:16.474043    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-794c96db68-55xsz" podStartSLOduration=2.474028018 podStartE2EDuration="2.474028018s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:30:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:30:16.473243371 +0000 UTC m=+328.943241517" watchObservedRunningTime="2026-09-16 17:30:16.474028018 +0000 UTC m=+328.944026152"
Sep 16 17:30:25.136720 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:30:25.136677    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-794c96db68-55xsz"
Sep 16 17:30:25.137121 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:30:25.136740    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-794c96db68-55xsz"
Sep 16 17:30:25.142113 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:30:25.142087    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-794c96db68-55xsz"
Sep 16 17:30:25.497344 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:30:25.497268    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-794c96db68-55xsz"
Sep 16 17:30:25.546491 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:30:25.546449    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5f7b9bb969-gdxqv"]
Sep 16 17:30:50.568316 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:30:50.568235    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5f7b9bb969-gdxqv" podUID="4020fe82-9799-424d-a7c0-9b9cad1eab40" containerName="console" containerID="cri-o://a62834b2ee9746f21f8eb26a604d38df6e3f932ef66a4bafa3972532d0a2f4a6" gracePeriod=15
Sep 16 17:30:50.812578 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:30:50.812550    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5f7b9bb969-gdxqv"
Sep 16 17:30:50.947734 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:30:50.947648    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b64nr\" (UniqueName: \"kubernetes.io/projected/4020fe82-9799-424d-a7c0-9b9cad1eab40-kube-api-access-b64nr\") pod \"4020fe82-9799-424d-a7c0-9b9cad1eab40\" (UID: \"4020fe82-9799-424d-a7c0-9b9cad1eab40\") "
Sep 16 17:30:50.947734 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:30:50.947700    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4020fe82-9799-424d-a7c0-9b9cad1eab40-service-ca\") pod \"4020fe82-9799-424d-a7c0-9b9cad1eab40\" (UID: \"4020fe82-9799-424d-a7c0-9b9cad1eab40\") "
Sep 16 17:30:50.947734 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:30:50.947717    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4020fe82-9799-424d-a7c0-9b9cad1eab40-trusted-ca-bundle\") pod \"4020fe82-9799-424d-a7c0-9b9cad1eab40\" (UID: \"4020fe82-9799-424d-a7c0-9b9cad1eab40\") "
Sep 16 17:30:50.948012 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:30:50.947791    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4020fe82-9799-424d-a7c0-9b9cad1eab40-oauth-serving-cert\") pod \"4020fe82-9799-424d-a7c0-9b9cad1eab40\" (UID: \"4020fe82-9799-424d-a7c0-9b9cad1eab40\") "
Sep 16 17:30:50.948012 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:30:50.947813    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4020fe82-9799-424d-a7c0-9b9cad1eab40-console-oauth-config\") pod \"4020fe82-9799-424d-a7c0-9b9cad1eab40\" (UID: \"4020fe82-9799-424d-a7c0-9b9cad1eab40\") "
Sep 16 17:30:50.948012 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:30:50.947870    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4020fe82-9799-424d-a7c0-9b9cad1eab40-console-serving-cert\") pod \"4020fe82-9799-424d-a7c0-9b9cad1eab40\" (UID: \"4020fe82-9799-424d-a7c0-9b9cad1eab40\") "
Sep 16 17:30:50.948012 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:30:50.947901    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4020fe82-9799-424d-a7c0-9b9cad1eab40-console-config\") pod \"4020fe82-9799-424d-a7c0-9b9cad1eab40\" (UID: \"4020fe82-9799-424d-a7c0-9b9cad1eab40\") "
Sep 16 17:30:50.948327 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:30:50.948228    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4020fe82-9799-424d-a7c0-9b9cad1eab40-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "4020fe82-9799-424d-a7c0-9b9cad1eab40" (UID: "4020fe82-9799-424d-a7c0-9b9cad1eab40"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 17:30:50.948404 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:30:50.948237    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4020fe82-9799-424d-a7c0-9b9cad1eab40-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "4020fe82-9799-424d-a7c0-9b9cad1eab40" (UID: "4020fe82-9799-424d-a7c0-9b9cad1eab40"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 17:30:50.948404 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:30:50.948374    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4020fe82-9799-424d-a7c0-9b9cad1eab40-console-config" (OuterVolumeSpecName: "console-config") pod "4020fe82-9799-424d-a7c0-9b9cad1eab40" (UID: "4020fe82-9799-424d-a7c0-9b9cad1eab40"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 17:30:50.948514 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:30:50.948428    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4020fe82-9799-424d-a7c0-9b9cad1eab40-service-ca" (OuterVolumeSpecName: "service-ca") pod "4020fe82-9799-424d-a7c0-9b9cad1eab40" (UID: "4020fe82-9799-424d-a7c0-9b9cad1eab40"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 17:30:50.949977 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:30:50.949950    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4020fe82-9799-424d-a7c0-9b9cad1eab40-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "4020fe82-9799-424d-a7c0-9b9cad1eab40" (UID: "4020fe82-9799-424d-a7c0-9b9cad1eab40"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 17:30:50.950071 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:30:50.950016    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4020fe82-9799-424d-a7c0-9b9cad1eab40-kube-api-access-b64nr" (OuterVolumeSpecName: "kube-api-access-b64nr") pod "4020fe82-9799-424d-a7c0-9b9cad1eab40" (UID: "4020fe82-9799-424d-a7c0-9b9cad1eab40"). InnerVolumeSpecName "kube-api-access-b64nr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 17:30:50.950071 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:30:50.950026    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4020fe82-9799-424d-a7c0-9b9cad1eab40-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "4020fe82-9799-424d-a7c0-9b9cad1eab40" (UID: "4020fe82-9799-424d-a7c0-9b9cad1eab40"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 17:30:51.048799 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:30:51.048761    2579 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4020fe82-9799-424d-a7c0-9b9cad1eab40-oauth-serving-cert\") on node \"ip-10-0-138-73.ec2.internal\" DevicePath \"\""
Sep 16 17:30:51.048799 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:30:51.048791    2579 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4020fe82-9799-424d-a7c0-9b9cad1eab40-console-oauth-config\") on node \"ip-10-0-138-73.ec2.internal\" DevicePath \"\""
Sep 16 17:30:51.048799 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:30:51.048801    2579 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4020fe82-9799-424d-a7c0-9b9cad1eab40-console-serving-cert\") on node \"ip-10-0-138-73.ec2.internal\" DevicePath \"\""
Sep 16 17:30:51.048799 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:30:51.048810    2579 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4020fe82-9799-424d-a7c0-9b9cad1eab40-console-config\") on node \"ip-10-0-138-73.ec2.internal\" DevicePath \"\""
Sep 16 17:30:51.049062 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:30:51.048818    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b64nr\" (UniqueName: \"kubernetes.io/projected/4020fe82-9799-424d-a7c0-9b9cad1eab40-kube-api-access-b64nr\") on node \"ip-10-0-138-73.ec2.internal\" DevicePath \"\""
Sep 16 17:30:51.049062 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:30:51.048826    2579 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4020fe82-9799-424d-a7c0-9b9cad1eab40-service-ca\") on node \"ip-10-0-138-73.ec2.internal\" DevicePath \"\""
Sep 16 17:30:51.049062 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:30:51.048836    2579 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4020fe82-9799-424d-a7c0-9b9cad1eab40-trusted-ca-bundle\") on node \"ip-10-0-138-73.ec2.internal\" DevicePath \"\""
Sep 16 17:30:51.602132 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:30:51.602095    2579 generic.go:356] "Generic (PLEG): container finished" podID="4020fe82-9799-424d-a7c0-9b9cad1eab40" containerID="a62834b2ee9746f21f8eb26a604d38df6e3f932ef66a4bafa3972532d0a2f4a6" exitCode=0
Sep 16 17:30:51.602585 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:30:51.602147    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5f7b9bb969-gdxqv"
Sep 16 17:30:51.602585 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:30:51.602187    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5f7b9bb969-gdxqv" event={"ID":"4020fe82-9799-424d-a7c0-9b9cad1eab40","Type":"ContainerDied","Data":"a62834b2ee9746f21f8eb26a604d38df6e3f932ef66a4bafa3972532d0a2f4a6"}
Sep 16 17:30:51.602585 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:30:51.602251    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5f7b9bb969-gdxqv" event={"ID":"4020fe82-9799-424d-a7c0-9b9cad1eab40","Type":"ContainerDied","Data":"7b0a4f580574dc0eac2201edd422fe789da7932d23e3bc6d3fda9e92a0cc547b"}
Sep 16 17:30:51.602585 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:30:51.602284    2579 scope.go:117] "RemoveContainer" containerID="a62834b2ee9746f21f8eb26a604d38df6e3f932ef66a4bafa3972532d0a2f4a6"
Sep 16 17:30:51.611466 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:30:51.611448    2579 scope.go:117] "RemoveContainer" containerID="a62834b2ee9746f21f8eb26a604d38df6e3f932ef66a4bafa3972532d0a2f4a6"
Sep 16 17:30:51.611726 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:30:51.611707    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a62834b2ee9746f21f8eb26a604d38df6e3f932ef66a4bafa3972532d0a2f4a6\": container with ID starting with a62834b2ee9746f21f8eb26a604d38df6e3f932ef66a4bafa3972532d0a2f4a6 not found: ID does not exist" containerID="a62834b2ee9746f21f8eb26a604d38df6e3f932ef66a4bafa3972532d0a2f4a6"
Sep 16 17:30:51.611782 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:30:51.611734    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a62834b2ee9746f21f8eb26a604d38df6e3f932ef66a4bafa3972532d0a2f4a6"} err="failed to get container status \"a62834b2ee9746f21f8eb26a604d38df6e3f932ef66a4bafa3972532d0a2f4a6\": rpc error: code = NotFound desc = could not find container \"a62834b2ee9746f21f8eb26a604d38df6e3f932ef66a4bafa3972532d0a2f4a6\": container with ID starting with a62834b2ee9746f21f8eb26a604d38df6e3f932ef66a4bafa3972532d0a2f4a6 not found: ID does not exist"
Sep 16 17:30:51.623187 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:30:51.623163    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5f7b9bb969-gdxqv"]
Sep 16 17:30:51.629058 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:30:51.629037    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5f7b9bb969-gdxqv"]
Sep 16 17:30:52.098169 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:30:52.098141    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4020fe82-9799-424d-a7c0-9b9cad1eab40" path="/var/lib/kubelet/pods/4020fe82-9799-424d-a7c0-9b9cad1eab40/volumes"
Sep 16 17:32:29.470855 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:32:29.470823    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7784ccdc77-wdvpr"]
Sep 16 17:32:29.471249 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:32:29.471181    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4020fe82-9799-424d-a7c0-9b9cad1eab40" containerName="console"
Sep 16 17:32:29.471249 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:32:29.471192    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="4020fe82-9799-424d-a7c0-9b9cad1eab40" containerName="console"
Sep 16 17:32:29.471249 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:32:29.471249    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="4020fe82-9799-424d-a7c0-9b9cad1eab40" containerName="console"
Sep 16 17:32:29.474083 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:32:29.474066    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7784ccdc77-wdvpr"
Sep 16 17:32:29.517293 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:32:29.517260    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7784ccdc77-wdvpr"]
Sep 16 17:32:29.551918 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:32:29.551879    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/30e17757-071e-4e55-a611-89f630626cd2-console-serving-cert\") pod \"console-7784ccdc77-wdvpr\" (UID: \"30e17757-071e-4e55-a611-89f630626cd2\") " pod="openshift-console/console-7784ccdc77-wdvpr"
Sep 16 17:32:29.552090 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:32:29.551929    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/30e17757-071e-4e55-a611-89f630626cd2-console-oauth-config\") pod \"console-7784ccdc77-wdvpr\" (UID: \"30e17757-071e-4e55-a611-89f630626cd2\") " pod="openshift-console/console-7784ccdc77-wdvpr"
Sep 16 17:32:29.552090 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:32:29.551949    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/30e17757-071e-4e55-a611-89f630626cd2-oauth-serving-cert\") pod \"console-7784ccdc77-wdvpr\" (UID: \"30e17757-071e-4e55-a611-89f630626cd2\") " pod="openshift-console/console-7784ccdc77-wdvpr"
Sep 16 17:32:29.552090 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:32:29.551965    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sswtn\" (UniqueName: \"kubernetes.io/projected/30e17757-071e-4e55-a611-89f630626cd2-kube-api-access-sswtn\") pod \"console-7784ccdc77-wdvpr\" (UID: \"30e17757-071e-4e55-a611-89f630626cd2\") " pod="openshift-console/console-7784ccdc77-wdvpr"
Sep 16 17:32:29.552090 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:32:29.551995    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/30e17757-071e-4e55-a611-89f630626cd2-console-config\") pod \"console-7784ccdc77-wdvpr\" (UID: \"30e17757-071e-4e55-a611-89f630626cd2\") " pod="openshift-console/console-7784ccdc77-wdvpr"
Sep 16 17:32:29.552090 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:32:29.552027    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/30e17757-071e-4e55-a611-89f630626cd2-service-ca\") pod \"console-7784ccdc77-wdvpr\" (UID: \"30e17757-071e-4e55-a611-89f630626cd2\") " pod="openshift-console/console-7784ccdc77-wdvpr"
Sep 16 17:32:29.552090 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:32:29.552051    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/30e17757-071e-4e55-a611-89f630626cd2-trusted-ca-bundle\") pod \"console-7784ccdc77-wdvpr\" (UID: \"30e17757-071e-4e55-a611-89f630626cd2\") " pod="openshift-console/console-7784ccdc77-wdvpr"
Sep 16 17:32:29.653105 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:32:29.653065    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/30e17757-071e-4e55-a611-89f630626cd2-console-config\") pod \"console-7784ccdc77-wdvpr\" (UID: \"30e17757-071e-4e55-a611-89f630626cd2\") " pod="openshift-console/console-7784ccdc77-wdvpr"
Sep 16 17:32:29.653105 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:32:29.653112    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/30e17757-071e-4e55-a611-89f630626cd2-service-ca\") pod \"console-7784ccdc77-wdvpr\" (UID: \"30e17757-071e-4e55-a611-89f630626cd2\") " pod="openshift-console/console-7784ccdc77-wdvpr"
Sep 16 17:32:29.653319 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:32:29.653227    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/30e17757-071e-4e55-a611-89f630626cd2-trusted-ca-bundle\") pod \"console-7784ccdc77-wdvpr\" (UID: \"30e17757-071e-4e55-a611-89f630626cd2\") " pod="openshift-console/console-7784ccdc77-wdvpr"
Sep 16 17:32:29.653319 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:32:29.653285    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/30e17757-071e-4e55-a611-89f630626cd2-console-serving-cert\") pod \"console-7784ccdc77-wdvpr\" (UID: \"30e17757-071e-4e55-a611-89f630626cd2\") " pod="openshift-console/console-7784ccdc77-wdvpr"
Sep 16 17:32:29.653389 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:32:29.653325    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/30e17757-071e-4e55-a611-89f630626cd2-console-oauth-config\") pod \"console-7784ccdc77-wdvpr\" (UID: \"30e17757-071e-4e55-a611-89f630626cd2\") " pod="openshift-console/console-7784ccdc77-wdvpr"
Sep 16 17:32:29.653389 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:32:29.653348    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/30e17757-071e-4e55-a611-89f630626cd2-oauth-serving-cert\") pod \"console-7784ccdc77-wdvpr\" (UID: \"30e17757-071e-4e55-a611-89f630626cd2\") " pod="openshift-console/console-7784ccdc77-wdvpr"
Sep 16 17:32:29.653389 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:32:29.653368    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sswtn\" (UniqueName: \"kubernetes.io/projected/30e17757-071e-4e55-a611-89f630626cd2-kube-api-access-sswtn\") pod \"console-7784ccdc77-wdvpr\" (UID: \"30e17757-071e-4e55-a611-89f630626cd2\") " pod="openshift-console/console-7784ccdc77-wdvpr"
Sep 16 17:32:29.653911 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:32:29.653879    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/30e17757-071e-4e55-a611-89f630626cd2-service-ca\") pod \"console-7784ccdc77-wdvpr\" (UID: \"30e17757-071e-4e55-a611-89f630626cd2\") " pod="openshift-console/console-7784ccdc77-wdvpr"
Sep 16 17:32:29.654030 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:32:29.653945    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/30e17757-071e-4e55-a611-89f630626cd2-console-config\") pod \"console-7784ccdc77-wdvpr\" (UID: \"30e17757-071e-4e55-a611-89f630626cd2\") " pod="openshift-console/console-7784ccdc77-wdvpr"
Sep 16 17:32:29.654100 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:32:29.654033    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/30e17757-071e-4e55-a611-89f630626cd2-oauth-serving-cert\") pod \"console-7784ccdc77-wdvpr\" (UID: \"30e17757-071e-4e55-a611-89f630626cd2\") " pod="openshift-console/console-7784ccdc77-wdvpr"
Sep 16 17:32:29.654100 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:32:29.654071    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/30e17757-071e-4e55-a611-89f630626cd2-trusted-ca-bundle\") pod \"console-7784ccdc77-wdvpr\" (UID: \"30e17757-071e-4e55-a611-89f630626cd2\") " pod="openshift-console/console-7784ccdc77-wdvpr"
Sep 16 17:32:29.655807 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:32:29.655785    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/30e17757-071e-4e55-a611-89f630626cd2-console-serving-cert\") pod \"console-7784ccdc77-wdvpr\" (UID: \"30e17757-071e-4e55-a611-89f630626cd2\") " pod="openshift-console/console-7784ccdc77-wdvpr"
Sep 16 17:32:29.655807 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:32:29.655803    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/30e17757-071e-4e55-a611-89f630626cd2-console-oauth-config\") pod \"console-7784ccdc77-wdvpr\" (UID: \"30e17757-071e-4e55-a611-89f630626cd2\") " pod="openshift-console/console-7784ccdc77-wdvpr"
Sep 16 17:32:29.663084 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:32:29.663064    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sswtn\" (UniqueName: \"kubernetes.io/projected/30e17757-071e-4e55-a611-89f630626cd2-kube-api-access-sswtn\") pod \"console-7784ccdc77-wdvpr\" (UID: \"30e17757-071e-4e55-a611-89f630626cd2\") " pod="openshift-console/console-7784ccdc77-wdvpr"
Sep 16 17:32:29.783226 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:32:29.783199    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7784ccdc77-wdvpr"
Sep 16 17:32:30.063315 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:32:30.063263    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7784ccdc77-wdvpr"]
Sep 16 17:32:30.065555 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:32:30.065528    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30e17757_071e_4e55_a611_89f630626cd2.slice/crio-ffc2d8b74f13c59b58faa2969a368070c38ac7cab9c0bc7af2fff431f022fc14 WatchSource:0}: Error finding container ffc2d8b74f13c59b58faa2969a368070c38ac7cab9c0bc7af2fff431f022fc14: Status 404 returned error can't find the container with id ffc2d8b74f13c59b58faa2969a368070c38ac7cab9c0bc7af2fff431f022fc14
Sep 16 17:32:31.004337 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:32:31.004306    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7784ccdc77-wdvpr" event={"ID":"30e17757-071e-4e55-a611-89f630626cd2","Type":"ContainerStarted","Data":"a88774150dab1f6ff1c11f825159db1dca0afd44502bac4252c31da5dcfb703c"}
Sep 16 17:32:31.004818 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:32:31.004353    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7784ccdc77-wdvpr" event={"ID":"30e17757-071e-4e55-a611-89f630626cd2","Type":"ContainerStarted","Data":"ffc2d8b74f13c59b58faa2969a368070c38ac7cab9c0bc7af2fff431f022fc14"}
Sep 16 17:32:31.030197 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:32:31.030132    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7784ccdc77-wdvpr" podStartSLOduration=2.030111529 podStartE2EDuration="2.030111529s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:32:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:32:31.028508412 +0000 UTC m=+463.498506546" watchObservedRunningTime="2026-09-16 17:32:31.030111529 +0000 UTC m=+463.500109686"
Sep 16 17:32:39.783776 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:32:39.783736    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7784ccdc77-wdvpr"
Sep 16 17:32:39.784186 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:32:39.783825    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7784ccdc77-wdvpr"
Sep 16 17:32:39.788551 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:32:39.788520    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7784ccdc77-wdvpr"
Sep 16 17:32:40.048268 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:32:40.048192    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7784ccdc77-wdvpr"
Sep 16 17:32:40.114812 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:32:40.114781    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-794c96db68-55xsz"]
Sep 16 17:32:59.753390 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:32:59.753348    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1akvsl2"]
Sep 16 17:32:59.757047 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:32:59.757024    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1akvsl2"
Sep 16 17:32:59.759824 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:32:59.759804    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 16 17:32:59.759976 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:32:59.759955    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-d2zmj\""
Sep 16 17:32:59.760834 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:32:59.760819    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 16 17:32:59.771564 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:32:59.771537    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1akvsl2"]
Sep 16 17:32:59.817344 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:32:59.817312    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/db390b49-8de5-441d-8ffe-21368d4a304f-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1akvsl2\" (UID: \"db390b49-8de5-441d-8ffe-21368d4a304f\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1akvsl2"
Sep 16 17:32:59.817514 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:32:59.817413    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/db390b49-8de5-441d-8ffe-21368d4a304f-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1akvsl2\" (UID: \"db390b49-8de5-441d-8ffe-21368d4a304f\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1akvsl2"
Sep 16 17:32:59.817514 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:32:59.817438    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jh2ll\" (UniqueName: \"kubernetes.io/projected/db390b49-8de5-441d-8ffe-21368d4a304f-kube-api-access-jh2ll\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1akvsl2\" (UID: \"db390b49-8de5-441d-8ffe-21368d4a304f\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1akvsl2"
Sep 16 17:32:59.918761 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:32:59.918722    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/db390b49-8de5-441d-8ffe-21368d4a304f-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1akvsl2\" (UID: \"db390b49-8de5-441d-8ffe-21368d4a304f\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1akvsl2"
Sep 16 17:32:59.918761 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:32:59.918765    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jh2ll\" (UniqueName: \"kubernetes.io/projected/db390b49-8de5-441d-8ffe-21368d4a304f-kube-api-access-jh2ll\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1akvsl2\" (UID: \"db390b49-8de5-441d-8ffe-21368d4a304f\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1akvsl2"
Sep 16 17:32:59.919029 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:32:59.918794    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/db390b49-8de5-441d-8ffe-21368d4a304f-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1akvsl2\" (UID: \"db390b49-8de5-441d-8ffe-21368d4a304f\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1akvsl2"
Sep 16 17:32:59.919115 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:32:59.919095    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/db390b49-8de5-441d-8ffe-21368d4a304f-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1akvsl2\" (UID: \"db390b49-8de5-441d-8ffe-21368d4a304f\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1akvsl2"
Sep 16 17:32:59.919165 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:32:59.919128    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/db390b49-8de5-441d-8ffe-21368d4a304f-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1akvsl2\" (UID: \"db390b49-8de5-441d-8ffe-21368d4a304f\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1akvsl2"
Sep 16 17:32:59.927719 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:32:59.927686    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jh2ll\" (UniqueName: \"kubernetes.io/projected/db390b49-8de5-441d-8ffe-21368d4a304f-kube-api-access-jh2ll\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1akvsl2\" (UID: \"db390b49-8de5-441d-8ffe-21368d4a304f\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1akvsl2"
Sep 16 17:33:00.066168 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:33:00.066134    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1akvsl2"
Sep 16 17:33:00.338052 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:33:00.337981    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1akvsl2"]
Sep 16 17:33:00.341620 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:33:00.341591    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb390b49_8de5_441d_8ffe_21368d4a304f.slice/crio-924af10a163fe760d03543bf765141065c932e0bd581a12893f9811ff54760f6 WatchSource:0}: Error finding container 924af10a163fe760d03543bf765141065c932e0bd581a12893f9811ff54760f6: Status 404 returned error can't find the container with id 924af10a163fe760d03543bf765141065c932e0bd581a12893f9811ff54760f6
Sep 16 17:33:00.343753 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:33:00.343737    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 17:33:01.130884 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:33:01.130825    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1akvsl2" event={"ID":"db390b49-8de5-441d-8ffe-21368d4a304f","Type":"ContainerStarted","Data":"924af10a163fe760d03543bf765141065c932e0bd581a12893f9811ff54760f6"}
Sep 16 17:33:05.136121 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:33:05.136070    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-794c96db68-55xsz" podUID="528737a5-a963-4703-bc0e-edd77943a7e7" containerName="console" containerID="cri-o://8eadd0f429e5ac5babc6b07d2f51b1e16bf7271328a32b13cd32c6f597e9a142" gracePeriod=15
Sep 16 17:33:05.494007 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:33:05.493924    2579 patch_prober.go:28] interesting pod/console-794c96db68-55xsz container/console namespace/openshift-console: Readiness probe status=failure output="Get \"https://10.134.0.27:8443/health\": dial tcp 10.134.0.27:8443: connect: connection refused" start-of-body=
Sep 16 17:33:05.494163 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:33:05.493998    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console/console-794c96db68-55xsz" podUID="528737a5-a963-4703-bc0e-edd77943a7e7" containerName="console" probeResult="failure" output="Get \"https://10.134.0.27:8443/health\": dial tcp 10.134.0.27:8443: connect: connection refused"
Sep 16 17:33:07.002056 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:33:07.002031    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-794c96db68-55xsz"
Sep 16 17:33:07.080990 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:33:07.080962    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/528737a5-a963-4703-bc0e-edd77943a7e7-console-serving-cert\") pod \"528737a5-a963-4703-bc0e-edd77943a7e7\" (UID: \"528737a5-a963-4703-bc0e-edd77943a7e7\") "
Sep 16 17:33:07.081104 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:33:07.081000    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/528737a5-a963-4703-bc0e-edd77943a7e7-trusted-ca-bundle\") pod \"528737a5-a963-4703-bc0e-edd77943a7e7\" (UID: \"528737a5-a963-4703-bc0e-edd77943a7e7\") "
Sep 16 17:33:07.081104 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:33:07.081023    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/528737a5-a963-4703-bc0e-edd77943a7e7-oauth-serving-cert\") pod \"528737a5-a963-4703-bc0e-edd77943a7e7\" (UID: \"528737a5-a963-4703-bc0e-edd77943a7e7\") "
Sep 16 17:33:07.081104 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:33:07.081046    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/528737a5-a963-4703-bc0e-edd77943a7e7-console-config\") pod \"528737a5-a963-4703-bc0e-edd77943a7e7\" (UID: \"528737a5-a963-4703-bc0e-edd77943a7e7\") "
Sep 16 17:33:07.081104 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:33:07.081101    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/528737a5-a963-4703-bc0e-edd77943a7e7-service-ca\") pod \"528737a5-a963-4703-bc0e-edd77943a7e7\" (UID: \"528737a5-a963-4703-bc0e-edd77943a7e7\") "
Sep 16 17:33:07.081284 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:33:07.081120    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tjzf4\" (UniqueName: \"kubernetes.io/projected/528737a5-a963-4703-bc0e-edd77943a7e7-kube-api-access-tjzf4\") pod \"528737a5-a963-4703-bc0e-edd77943a7e7\" (UID: \"528737a5-a963-4703-bc0e-edd77943a7e7\") "
Sep 16 17:33:07.081284 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:33:07.081148    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/528737a5-a963-4703-bc0e-edd77943a7e7-console-oauth-config\") pod \"528737a5-a963-4703-bc0e-edd77943a7e7\" (UID: \"528737a5-a963-4703-bc0e-edd77943a7e7\") "
Sep 16 17:33:07.081485 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:33:07.081441    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/528737a5-a963-4703-bc0e-edd77943a7e7-service-ca" (OuterVolumeSpecName: "service-ca") pod "528737a5-a963-4703-bc0e-edd77943a7e7" (UID: "528737a5-a963-4703-bc0e-edd77943a7e7"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 17:33:07.081485 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:33:07.081377    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/528737a5-a963-4703-bc0e-edd77943a7e7-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "528737a5-a963-4703-bc0e-edd77943a7e7" (UID: "528737a5-a963-4703-bc0e-edd77943a7e7"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 17:33:07.081665 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:33:07.081583    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/528737a5-a963-4703-bc0e-edd77943a7e7-console-config" (OuterVolumeSpecName: "console-config") pod "528737a5-a963-4703-bc0e-edd77943a7e7" (UID: "528737a5-a963-4703-bc0e-edd77943a7e7"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 17:33:07.081737 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:33:07.081717    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/528737a5-a963-4703-bc0e-edd77943a7e7-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "528737a5-a963-4703-bc0e-edd77943a7e7" (UID: "528737a5-a963-4703-bc0e-edd77943a7e7"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 17:33:07.083031 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:33:07.083003    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/528737a5-a963-4703-bc0e-edd77943a7e7-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "528737a5-a963-4703-bc0e-edd77943a7e7" (UID: "528737a5-a963-4703-bc0e-edd77943a7e7"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 17:33:07.083261 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:33:07.083238    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/528737a5-a963-4703-bc0e-edd77943a7e7-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "528737a5-a963-4703-bc0e-edd77943a7e7" (UID: "528737a5-a963-4703-bc0e-edd77943a7e7"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 17:33:07.083316 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:33:07.083293    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/528737a5-a963-4703-bc0e-edd77943a7e7-kube-api-access-tjzf4" (OuterVolumeSpecName: "kube-api-access-tjzf4") pod "528737a5-a963-4703-bc0e-edd77943a7e7" (UID: "528737a5-a963-4703-bc0e-edd77943a7e7"). InnerVolumeSpecName "kube-api-access-tjzf4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 17:33:07.164414 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:33:07.164323    2579 generic.go:356] "Generic (PLEG): container finished" podID="db390b49-8de5-441d-8ffe-21368d4a304f" containerID="a3a2c478c83e6557ed832153a77e4e9db0e5b48ec746c51639b152705ceff495" exitCode=0
Sep 16 17:33:07.164579 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:33:07.164404    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1akvsl2" event={"ID":"db390b49-8de5-441d-8ffe-21368d4a304f","Type":"ContainerDied","Data":"a3a2c478c83e6557ed832153a77e4e9db0e5b48ec746c51639b152705ceff495"}
Sep 16 17:33:07.165774 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:33:07.165739    2579 generic.go:356] "Generic (PLEG): container finished" podID="528737a5-a963-4703-bc0e-edd77943a7e7" containerID="8eadd0f429e5ac5babc6b07d2f51b1e16bf7271328a32b13cd32c6f597e9a142" exitCode=0
Sep 16 17:33:07.165901 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:33:07.165783    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-794c96db68-55xsz"
Sep 16 17:33:07.165901 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:33:07.165803    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-794c96db68-55xsz" event={"ID":"528737a5-a963-4703-bc0e-edd77943a7e7","Type":"ContainerDied","Data":"8eadd0f429e5ac5babc6b07d2f51b1e16bf7271328a32b13cd32c6f597e9a142"}
Sep 16 17:33:07.165901 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:33:07.165839    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-794c96db68-55xsz" event={"ID":"528737a5-a963-4703-bc0e-edd77943a7e7","Type":"ContainerDied","Data":"2ae9e78b15bc627ae02da011011d6a2a754f737b4b322039aefbfdf0aba1110f"}
Sep 16 17:33:07.165901 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:33:07.165873    2579 scope.go:117] "RemoveContainer" containerID="8eadd0f429e5ac5babc6b07d2f51b1e16bf7271328a32b13cd32c6f597e9a142"
Sep 16 17:33:07.174670 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:33:07.174654    2579 scope.go:117] "RemoveContainer" containerID="8eadd0f429e5ac5babc6b07d2f51b1e16bf7271328a32b13cd32c6f597e9a142"
Sep 16 17:33:07.174931 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:33:07.174913    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8eadd0f429e5ac5babc6b07d2f51b1e16bf7271328a32b13cd32c6f597e9a142\": container with ID starting with 8eadd0f429e5ac5babc6b07d2f51b1e16bf7271328a32b13cd32c6f597e9a142 not found: ID does not exist" containerID="8eadd0f429e5ac5babc6b07d2f51b1e16bf7271328a32b13cd32c6f597e9a142"
Sep 16 17:33:07.174986 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:33:07.174941    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8eadd0f429e5ac5babc6b07d2f51b1e16bf7271328a32b13cd32c6f597e9a142"} err="failed to get container status \"8eadd0f429e5ac5babc6b07d2f51b1e16bf7271328a32b13cd32c6f597e9a142\": rpc error: code = NotFound desc = could not find container \"8eadd0f429e5ac5babc6b07d2f51b1e16bf7271328a32b13cd32c6f597e9a142\": container with ID starting with 8eadd0f429e5ac5babc6b07d2f51b1e16bf7271328a32b13cd32c6f597e9a142 not found: ID does not exist"
Sep 16 17:33:07.182355 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:33:07.182334    2579 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/528737a5-a963-4703-bc0e-edd77943a7e7-trusted-ca-bundle\") on node \"ip-10-0-138-73.ec2.internal\" DevicePath \"\""
Sep 16 17:33:07.182355 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:33:07.182356    2579 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/528737a5-a963-4703-bc0e-edd77943a7e7-oauth-serving-cert\") on node \"ip-10-0-138-73.ec2.internal\" DevicePath \"\""
Sep 16 17:33:07.182458 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:33:07.182365    2579 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/528737a5-a963-4703-bc0e-edd77943a7e7-console-config\") on node \"ip-10-0-138-73.ec2.internal\" DevicePath \"\""
Sep 16 17:33:07.182458 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:33:07.182373    2579 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/528737a5-a963-4703-bc0e-edd77943a7e7-service-ca\") on node \"ip-10-0-138-73.ec2.internal\" DevicePath \"\""
Sep 16 17:33:07.182458 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:33:07.182380    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tjzf4\" (UniqueName: \"kubernetes.io/projected/528737a5-a963-4703-bc0e-edd77943a7e7-kube-api-access-tjzf4\") on node \"ip-10-0-138-73.ec2.internal\" DevicePath \"\""
Sep 16 17:33:07.182458 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:33:07.182388    2579 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/528737a5-a963-4703-bc0e-edd77943a7e7-console-oauth-config\") on node \"ip-10-0-138-73.ec2.internal\" DevicePath \"\""
Sep 16 17:33:07.182458 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:33:07.182396    2579 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/528737a5-a963-4703-bc0e-edd77943a7e7-console-serving-cert\") on node \"ip-10-0-138-73.ec2.internal\" DevicePath \"\""
Sep 16 17:33:07.195603 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:33:07.195579    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-794c96db68-55xsz"]
Sep 16 17:33:07.197205 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:33:07.197185    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-794c96db68-55xsz"]
Sep 16 17:33:08.100260 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:33:08.100222    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="528737a5-a963-4703-bc0e-edd77943a7e7" path="/var/lib/kubelet/pods/528737a5-a963-4703-bc0e-edd77943a7e7/volumes"
Sep 16 17:33:09.177693 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:33:09.177664    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1akvsl2" event={"ID":"db390b49-8de5-441d-8ffe-21368d4a304f","Type":"ContainerStarted","Data":"7e8dbe678a89cfa098af2d59430fed42ff8469fe18bf98d93c4d5e4f92b36e39"}
Sep 16 17:33:10.183992 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:33:10.183950    2579 generic.go:356] "Generic (PLEG): container finished" podID="db390b49-8de5-441d-8ffe-21368d4a304f" containerID="7e8dbe678a89cfa098af2d59430fed42ff8469fe18bf98d93c4d5e4f92b36e39" exitCode=0
Sep 16 17:33:10.184393 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:33:10.184043    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1akvsl2" event={"ID":"db390b49-8de5-441d-8ffe-21368d4a304f","Type":"ContainerDied","Data":"7e8dbe678a89cfa098af2d59430fed42ff8469fe18bf98d93c4d5e4f92b36e39"}
Sep 16 17:33:18.222199 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:33:18.222166    2579 generic.go:356] "Generic (PLEG): container finished" podID="db390b49-8de5-441d-8ffe-21368d4a304f" containerID="949288faba3a37ab5739aa7c8f5372cf3874b77414379cc1017a5a2a683cfe68" exitCode=0
Sep 16 17:33:18.222629 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:33:18.222259    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1akvsl2" event={"ID":"db390b49-8de5-441d-8ffe-21368d4a304f","Type":"ContainerDied","Data":"949288faba3a37ab5739aa7c8f5372cf3874b77414379cc1017a5a2a683cfe68"}
Sep 16 17:33:19.349528 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:33:19.349503    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1akvsl2"
Sep 16 17:33:19.489186 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:33:19.489083    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/db390b49-8de5-441d-8ffe-21368d4a304f-bundle\") pod \"db390b49-8de5-441d-8ffe-21368d4a304f\" (UID: \"db390b49-8de5-441d-8ffe-21368d4a304f\") "
Sep 16 17:33:19.489186 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:33:19.489133    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jh2ll\" (UniqueName: \"kubernetes.io/projected/db390b49-8de5-441d-8ffe-21368d4a304f-kube-api-access-jh2ll\") pod \"db390b49-8de5-441d-8ffe-21368d4a304f\" (UID: \"db390b49-8de5-441d-8ffe-21368d4a304f\") "
Sep 16 17:33:19.489409 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:33:19.489231    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/db390b49-8de5-441d-8ffe-21368d4a304f-util\") pod \"db390b49-8de5-441d-8ffe-21368d4a304f\" (UID: \"db390b49-8de5-441d-8ffe-21368d4a304f\") "
Sep 16 17:33:19.489811 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:33:19.489786    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db390b49-8de5-441d-8ffe-21368d4a304f-bundle" (OuterVolumeSpecName: "bundle") pod "db390b49-8de5-441d-8ffe-21368d4a304f" (UID: "db390b49-8de5-441d-8ffe-21368d4a304f"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:33:19.491375 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:33:19.491343    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db390b49-8de5-441d-8ffe-21368d4a304f-kube-api-access-jh2ll" (OuterVolumeSpecName: "kube-api-access-jh2ll") pod "db390b49-8de5-441d-8ffe-21368d4a304f" (UID: "db390b49-8de5-441d-8ffe-21368d4a304f"). InnerVolumeSpecName "kube-api-access-jh2ll". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 17:33:19.493369 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:33:19.493341    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db390b49-8de5-441d-8ffe-21368d4a304f-util" (OuterVolumeSpecName: "util") pod "db390b49-8de5-441d-8ffe-21368d4a304f" (UID: "db390b49-8de5-441d-8ffe-21368d4a304f"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:33:19.590688 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:33:19.590651    2579 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/db390b49-8de5-441d-8ffe-21368d4a304f-util\") on node \"ip-10-0-138-73.ec2.internal\" DevicePath \"\""
Sep 16 17:33:19.590688 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:33:19.590681    2579 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/db390b49-8de5-441d-8ffe-21368d4a304f-bundle\") on node \"ip-10-0-138-73.ec2.internal\" DevicePath \"\""
Sep 16 17:33:19.590688 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:33:19.590692    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jh2ll\" (UniqueName: \"kubernetes.io/projected/db390b49-8de5-441d-8ffe-21368d4a304f-kube-api-access-jh2ll\") on node \"ip-10-0-138-73.ec2.internal\" DevicePath \"\""
Sep 16 17:33:20.233211 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:33:20.233174    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1akvsl2" event={"ID":"db390b49-8de5-441d-8ffe-21368d4a304f","Type":"ContainerDied","Data":"924af10a163fe760d03543bf765141065c932e0bd581a12893f9811ff54760f6"}
Sep 16 17:33:20.233374 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:33:20.233229    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="924af10a163fe760d03543bf765141065c932e0bd581a12893f9811ff54760f6"
Sep 16 17:33:20.233374 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:33:20.233190    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1akvsl2"
Sep 16 17:33:24.973919 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:33:24.973883    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-8grlc"]
Sep 16 17:33:24.974281 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:33:24.974202    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="db390b49-8de5-441d-8ffe-21368d4a304f" containerName="extract"
Sep 16 17:33:24.974281 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:33:24.974214    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="db390b49-8de5-441d-8ffe-21368d4a304f" containerName="extract"
Sep 16 17:33:24.974281 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:33:24.974223    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="528737a5-a963-4703-bc0e-edd77943a7e7" containerName="console"
Sep 16 17:33:24.974281 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:33:24.974230    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="528737a5-a963-4703-bc0e-edd77943a7e7" containerName="console"
Sep 16 17:33:24.974281 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:33:24.974236    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="db390b49-8de5-441d-8ffe-21368d4a304f" containerName="util"
Sep 16 17:33:24.974281 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:33:24.974242    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="db390b49-8de5-441d-8ffe-21368d4a304f" containerName="util"
Sep 16 17:33:24.974281 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:33:24.974256    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="db390b49-8de5-441d-8ffe-21368d4a304f" containerName="pull"
Sep 16 17:33:24.974281 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:33:24.974261    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="db390b49-8de5-441d-8ffe-21368d4a304f" containerName="pull"
Sep 16 17:33:24.974535 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:33:24.974320    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="db390b49-8de5-441d-8ffe-21368d4a304f" containerName="extract"
Sep 16 17:33:24.974535 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:33:24.974329    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="528737a5-a963-4703-bc0e-edd77943a7e7" containerName="console"
Sep 16 17:33:25.007607 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:33:25.007577    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-8grlc"
Sep 16 17:33:25.015927 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:33:25.015902    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-8grlc"]
Sep 16 17:33:25.016759 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:33:25.016744    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
Sep 16 17:33:25.017067 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:33:25.017046    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"custom-metrics-autoscaler-operator-dockercfg-7qsh9\""
Sep 16 17:33:25.017184 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:33:25.017164    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Sep 16 17:33:25.017764 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:33:25.017741    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Sep 16 17:33:25.138272 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:33:25.138234    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zs9fg\" (UniqueName: \"kubernetes.io/projected/e310aa51-794f-4e68-8467-9d000d825e75-kube-api-access-zs9fg\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-8grlc\" (UID: \"e310aa51-794f-4e68-8467-9d000d825e75\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-8grlc"
Sep 16 17:33:25.138436 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:33:25.138300    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/e310aa51-794f-4e68-8467-9d000d825e75-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-8grlc\" (UID: \"e310aa51-794f-4e68-8467-9d000d825e75\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-8grlc"
Sep 16 17:33:25.239349 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:33:25.239263    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zs9fg\" (UniqueName: \"kubernetes.io/projected/e310aa51-794f-4e68-8467-9d000d825e75-kube-api-access-zs9fg\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-8grlc\" (UID: \"e310aa51-794f-4e68-8467-9d000d825e75\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-8grlc"
Sep 16 17:33:25.239349 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:33:25.239328    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/e310aa51-794f-4e68-8467-9d000d825e75-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-8grlc\" (UID: \"e310aa51-794f-4e68-8467-9d000d825e75\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-8grlc"
Sep 16 17:33:25.241660 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:33:25.241630    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/e310aa51-794f-4e68-8467-9d000d825e75-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-8grlc\" (UID: \"e310aa51-794f-4e68-8467-9d000d825e75\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-8grlc"
Sep 16 17:33:25.254894 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:33:25.254860    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zs9fg\" (UniqueName: \"kubernetes.io/projected/e310aa51-794f-4e68-8467-9d000d825e75-kube-api-access-zs9fg\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-8grlc\" (UID: \"e310aa51-794f-4e68-8467-9d000d825e75\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-8grlc"
Sep 16 17:33:25.319006 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:33:25.318972    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-8grlc"
Sep 16 17:33:25.622824 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:33:25.622785    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-8grlc"]
Sep 16 17:33:25.628589 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:33:25.628561    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode310aa51_794f_4e68_8467_9d000d825e75.slice/crio-685574a2f2d520f16474b6e373a1cd84a36e639e1d3d6008b96d90462f8ecf6c WatchSource:0}: Error finding container 685574a2f2d520f16474b6e373a1cd84a36e639e1d3d6008b96d90462f8ecf6c: Status 404 returned error can't find the container with id 685574a2f2d520f16474b6e373a1cd84a36e639e1d3d6008b96d90462f8ecf6c
Sep 16 17:33:26.261120 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:33:26.261085    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-8grlc" event={"ID":"e310aa51-794f-4e68-8467-9d000d825e75","Type":"ContainerStarted","Data":"685574a2f2d520f16474b6e373a1cd84a36e639e1d3d6008b96d90462f8ecf6c"}
Sep 16 17:33:29.283066 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:33:29.283031    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-8grlc" event={"ID":"e310aa51-794f-4e68-8467-9d000d825e75","Type":"ContainerStarted","Data":"da3348ad5eda0569c090a6797d38e1f82f620d81afccc9337243a5dfea56938f"}
Sep 16 17:33:29.283502 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:33:29.283108    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-8grlc"
Sep 16 17:33:29.338453 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:33:29.338400    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-8grlc" podStartSLOduration=2.592375765 podStartE2EDuration="5.338383344s" totalImagesPullingTime="2.746007579s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:33:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:33:29.32307936 +0000 UTC m=+521.793077492" watchObservedRunningTime="2026-09-16 17:33:29.338383344 +0000 UTC m=+521.808381479"
Sep 16 17:33:29.676955 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:33:29.676861    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-operator-579c8958f7-lzdrn"]
Sep 16 17:33:29.679383 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:33:29.679365    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-579c8958f7-lzdrn"
Sep 16 17:33:29.682053 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:33:29.682026    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Sep 16 17:33:29.682206 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:33:29.682188    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-q9dmd\""
Sep 16 17:33:29.682386 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:33:29.682372    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-certs\""
Sep 16 17:33:29.715810 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:33:29.715782    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-579c8958f7-lzdrn"]
Sep 16 17:33:29.783069 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:33:29.783036    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kt2kc\" (UniqueName: \"kubernetes.io/projected/d6688351-3ab6-4c06-992a-725c16c75dbf-kube-api-access-kt2kc\") pod \"keda-operator-579c8958f7-lzdrn\" (UID: \"d6688351-3ab6-4c06-992a-725c16c75dbf\") " pod="openshift-keda/keda-operator-579c8958f7-lzdrn"
Sep 16 17:33:29.783303 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:33:29.783090    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/d6688351-3ab6-4c06-992a-725c16c75dbf-cabundle0\") pod \"keda-operator-579c8958f7-lzdrn\" (UID: \"d6688351-3ab6-4c06-992a-725c16c75dbf\") " pod="openshift-keda/keda-operator-579c8958f7-lzdrn"
Sep 16 17:33:29.783303 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:33:29.783174    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/d6688351-3ab6-4c06-992a-725c16c75dbf-certificates\") pod \"keda-operator-579c8958f7-lzdrn\" (UID: \"d6688351-3ab6-4c06-992a-725c16c75dbf\") " pod="openshift-keda/keda-operator-579c8958f7-lzdrn"
Sep 16 17:33:29.884480 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:33:29.884440    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/d6688351-3ab6-4c06-992a-725c16c75dbf-cabundle0\") pod \"keda-operator-579c8958f7-lzdrn\" (UID: \"d6688351-3ab6-4c06-992a-725c16c75dbf\") " pod="openshift-keda/keda-operator-579c8958f7-lzdrn"
Sep 16 17:33:29.884688 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:33:29.884499    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/d6688351-3ab6-4c06-992a-725c16c75dbf-certificates\") pod \"keda-operator-579c8958f7-lzdrn\" (UID: \"d6688351-3ab6-4c06-992a-725c16c75dbf\") " pod="openshift-keda/keda-operator-579c8958f7-lzdrn"
Sep 16 17:33:29.884688 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:33:29.884572    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kt2kc\" (UniqueName: \"kubernetes.io/projected/d6688351-3ab6-4c06-992a-725c16c75dbf-kube-api-access-kt2kc\") pod \"keda-operator-579c8958f7-lzdrn\" (UID: \"d6688351-3ab6-4c06-992a-725c16c75dbf\") " pod="openshift-keda/keda-operator-579c8958f7-lzdrn"
Sep 16 17:33:29.884688 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:33:29.884673    2579 secret.go:281] references non-existent secret key: ca.crt
Sep 16 17:33:29.884796 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:33:29.884692    2579 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
Sep 16 17:33:29.884796 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:33:29.884702    2579 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-579c8958f7-lzdrn: references non-existent secret key: ca.crt
Sep 16 17:33:29.884796 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:33:29.884777    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d6688351-3ab6-4c06-992a-725c16c75dbf-certificates podName:d6688351-3ab6-4c06-992a-725c16c75dbf nodeName:}" failed. No retries permitted until 2026-09-16 17:33:30.384760143 +0000 UTC m=+522.854758472 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/d6688351-3ab6-4c06-992a-725c16c75dbf-certificates") pod "keda-operator-579c8958f7-lzdrn" (UID: "d6688351-3ab6-4c06-992a-725c16c75dbf") : references non-existent secret key: ca.crt
Sep 16 17:33:29.885270 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:33:29.885245    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/d6688351-3ab6-4c06-992a-725c16c75dbf-cabundle0\") pod \"keda-operator-579c8958f7-lzdrn\" (UID: \"d6688351-3ab6-4c06-992a-725c16c75dbf\") " pod="openshift-keda/keda-operator-579c8958f7-lzdrn"
Sep 16 17:33:29.908262 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:33:29.908229    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kt2kc\" (UniqueName: \"kubernetes.io/projected/d6688351-3ab6-4c06-992a-725c16c75dbf-kube-api-access-kt2kc\") pod \"keda-operator-579c8958f7-lzdrn\" (UID: \"d6688351-3ab6-4c06-992a-725c16c75dbf\") " pod="openshift-keda/keda-operator-579c8958f7-lzdrn"
Sep 16 17:33:30.389461 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:33:30.389420    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/d6688351-3ab6-4c06-992a-725c16c75dbf-certificates\") pod \"keda-operator-579c8958f7-lzdrn\" (UID: \"d6688351-3ab6-4c06-992a-725c16c75dbf\") " pod="openshift-keda/keda-operator-579c8958f7-lzdrn"
Sep 16 17:33:30.392067 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:33:30.392043    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/d6688351-3ab6-4c06-992a-725c16c75dbf-certificates\") pod \"keda-operator-579c8958f7-lzdrn\" (UID: \"d6688351-3ab6-4c06-992a-725c16c75dbf\") " pod="openshift-keda/keda-operator-579c8958f7-lzdrn"
Sep 16 17:33:30.589181 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:33:30.589142    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-579c8958f7-lzdrn"
Sep 16 17:33:30.905882 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:33:30.905773    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-579c8958f7-lzdrn"]
Sep 16 17:33:30.910353 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:33:30.910318    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6688351_3ab6_4c06_992a_725c16c75dbf.slice/crio-60a28006a805d1078c5d9199c3503afaa5c358582cf34a9f1c2c68b6684cbb53 WatchSource:0}: Error finding container 60a28006a805d1078c5d9199c3503afaa5c358582cf34a9f1c2c68b6684cbb53: Status 404 returned error can't find the container with id 60a28006a805d1078c5d9199c3503afaa5c358582cf34a9f1c2c68b6684cbb53
Sep 16 17:33:31.292931 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:33:31.292892    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-579c8958f7-lzdrn" event={"ID":"d6688351-3ab6-4c06-992a-725c16c75dbf","Type":"ContainerStarted","Data":"60a28006a805d1078c5d9199c3503afaa5c358582cf34a9f1c2c68b6684cbb53"}
Sep 16 17:33:35.312596 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:33:35.312564    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-579c8958f7-lzdrn" event={"ID":"d6688351-3ab6-4c06-992a-725c16c75dbf","Type":"ContainerStarted","Data":"9d4d0d3873cbc0f8a4abc5bb8e576d5973cb5df689dfd591ad7a6632427faf3f"}
Sep 16 17:33:35.312978 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:33:35.312628    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-operator-579c8958f7-lzdrn"
Sep 16 17:33:35.356357 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:33:35.356297    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-operator-579c8958f7-lzdrn" podStartSLOduration=3.005626409 podStartE2EDuration="6.356282809s" totalImagesPullingTime="3.3506564s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:33:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:33:35.355395637 +0000 UTC m=+527.825393772" watchObservedRunningTime="2026-09-16 17:33:35.356282809 +0000 UTC m=+527.826280943"
Sep 16 17:33:50.290052 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:33:50.290018    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-8grlc"
Sep 16 17:33:56.320136 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:33:56.320103    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-operator-579c8958f7-lzdrn"
Sep 16 17:34:52.670985 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:34:52.670951    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-cc8594966-x2w6w"]
Sep 16 17:34:52.673580 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:34:52.673557    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-cc8594966-x2w6w"
Sep 16 17:34:52.676985 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:34:52.676962    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-webhook-server-cert\""
Sep 16 17:34:52.677100 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:34:52.676970    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 16 17:34:52.677100 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:34:52.676972    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 16 17:34:52.677804 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:34:52.677787    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-controller-manager-dockercfg-vmkqh\""
Sep 16 17:34:52.686966 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:34:52.686942    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-cc8594966-x2w6w"]
Sep 16 17:34:52.706354 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:34:52.706317    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-64568bcd49-gzwdn"]
Sep 16 17:34:52.708860 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:34:52.708826    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-gzwdn"
Sep 16 17:34:52.711290 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:34:52.711268    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Sep 16 17:34:52.711396 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:34:52.711357    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-4xwd9\""
Sep 16 17:34:52.722640 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:34:52.722615    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-gzwdn"]
Sep 16 17:34:52.785877 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:34:52.785811    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4n2jn\" (UniqueName: \"kubernetes.io/projected/5b6acb17-7da6-4c72-a34b-81c81dab2f0b-kube-api-access-4n2jn\") pod \"kserve-controller-manager-cc8594966-x2w6w\" (UID: \"5b6acb17-7da6-4c72-a34b-81c81dab2f0b\") " pod="kserve/kserve-controller-manager-cc8594966-x2w6w"
Sep 16 17:34:52.786091 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:34:52.785929    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/a9dc8c86-0e70-401e-b73e-65e7d9768032-data\") pod \"seaweedfs-64568bcd49-gzwdn\" (UID: \"a9dc8c86-0e70-401e-b73e-65e7d9768032\") " pod="kserve/seaweedfs-64568bcd49-gzwdn"
Sep 16 17:34:52.786091 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:34:52.786001    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljtjv\" (UniqueName: \"kubernetes.io/projected/a9dc8c86-0e70-401e-b73e-65e7d9768032-kube-api-access-ljtjv\") pod \"seaweedfs-64568bcd49-gzwdn\" (UID: \"a9dc8c86-0e70-401e-b73e-65e7d9768032\") " pod="kserve/seaweedfs-64568bcd49-gzwdn"
Sep 16 17:34:52.786091 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:34:52.786035    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5b6acb17-7da6-4c72-a34b-81c81dab2f0b-cert\") pod \"kserve-controller-manager-cc8594966-x2w6w\" (UID: \"5b6acb17-7da6-4c72-a34b-81c81dab2f0b\") " pod="kserve/kserve-controller-manager-cc8594966-x2w6w"
Sep 16 17:34:52.887498 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:34:52.887458    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5b6acb17-7da6-4c72-a34b-81c81dab2f0b-cert\") pod \"kserve-controller-manager-cc8594966-x2w6w\" (UID: \"5b6acb17-7da6-4c72-a34b-81c81dab2f0b\") " pod="kserve/kserve-controller-manager-cc8594966-x2w6w"
Sep 16 17:34:52.887684 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:34:52.887539    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4n2jn\" (UniqueName: \"kubernetes.io/projected/5b6acb17-7da6-4c72-a34b-81c81dab2f0b-kube-api-access-4n2jn\") pod \"kserve-controller-manager-cc8594966-x2w6w\" (UID: \"5b6acb17-7da6-4c72-a34b-81c81dab2f0b\") " pod="kserve/kserve-controller-manager-cc8594966-x2w6w"
Sep 16 17:34:52.887684 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:34:52.887572    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/a9dc8c86-0e70-401e-b73e-65e7d9768032-data\") pod \"seaweedfs-64568bcd49-gzwdn\" (UID: \"a9dc8c86-0e70-401e-b73e-65e7d9768032\") " pod="kserve/seaweedfs-64568bcd49-gzwdn"
Sep 16 17:34:52.887684 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:34:52.887609    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljtjv\" (UniqueName: \"kubernetes.io/projected/a9dc8c86-0e70-401e-b73e-65e7d9768032-kube-api-access-ljtjv\") pod \"seaweedfs-64568bcd49-gzwdn\" (UID: \"a9dc8c86-0e70-401e-b73e-65e7d9768032\") " pod="kserve/seaweedfs-64568bcd49-gzwdn"
Sep 16 17:34:52.887684 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:34:52.887618    2579 secret.go:189] Couldn't get secret kserve/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found
Sep 16 17:34:52.887949 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:34:52.887699    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b6acb17-7da6-4c72-a34b-81c81dab2f0b-cert podName:5b6acb17-7da6-4c72-a34b-81c81dab2f0b nodeName:}" failed. No retries permitted until 2026-09-16 17:34:53.387677632 +0000 UTC m=+605.857675745 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/5b6acb17-7da6-4c72-a34b-81c81dab2f0b-cert") pod "kserve-controller-manager-cc8594966-x2w6w" (UID: "5b6acb17-7da6-4c72-a34b-81c81dab2f0b") : secret "kserve-webhook-server-cert" not found
Sep 16 17:34:52.888063 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:34:52.888045    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/a9dc8c86-0e70-401e-b73e-65e7d9768032-data\") pod \"seaweedfs-64568bcd49-gzwdn\" (UID: \"a9dc8c86-0e70-401e-b73e-65e7d9768032\") " pod="kserve/seaweedfs-64568bcd49-gzwdn"
Sep 16 17:34:52.896209 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:34:52.896185    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljtjv\" (UniqueName: \"kubernetes.io/projected/a9dc8c86-0e70-401e-b73e-65e7d9768032-kube-api-access-ljtjv\") pod \"seaweedfs-64568bcd49-gzwdn\" (UID: \"a9dc8c86-0e70-401e-b73e-65e7d9768032\") " pod="kserve/seaweedfs-64568bcd49-gzwdn"
Sep 16 17:34:52.902381 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:34:52.902360    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4n2jn\" (UniqueName: \"kubernetes.io/projected/5b6acb17-7da6-4c72-a34b-81c81dab2f0b-kube-api-access-4n2jn\") pod \"kserve-controller-manager-cc8594966-x2w6w\" (UID: \"5b6acb17-7da6-4c72-a34b-81c81dab2f0b\") " pod="kserve/kserve-controller-manager-cc8594966-x2w6w"
Sep 16 17:34:53.018568 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:34:53.018477    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-gzwdn"
Sep 16 17:34:53.291282 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:34:53.291215    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-gzwdn"]
Sep 16 17:34:53.294228 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:34:53.294191    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9dc8c86_0e70_401e_b73e_65e7d9768032.slice/crio-46ea692db8317e5726399d4a2b341257bceb8237616ad8ee6d80f9d024fe5248 WatchSource:0}: Error finding container 46ea692db8317e5726399d4a2b341257bceb8237616ad8ee6d80f9d024fe5248: Status 404 returned error can't find the container with id 46ea692db8317e5726399d4a2b341257bceb8237616ad8ee6d80f9d024fe5248
Sep 16 17:34:53.392691 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:34:53.392651    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5b6acb17-7da6-4c72-a34b-81c81dab2f0b-cert\") pod \"kserve-controller-manager-cc8594966-x2w6w\" (UID: \"5b6acb17-7da6-4c72-a34b-81c81dab2f0b\") " pod="kserve/kserve-controller-manager-cc8594966-x2w6w"
Sep 16 17:34:53.395114 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:34:53.395095    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5b6acb17-7da6-4c72-a34b-81c81dab2f0b-cert\") pod \"kserve-controller-manager-cc8594966-x2w6w\" (UID: \"5b6acb17-7da6-4c72-a34b-81c81dab2f0b\") " pod="kserve/kserve-controller-manager-cc8594966-x2w6w"
Sep 16 17:34:53.583970 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:34:53.583876    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-cc8594966-x2w6w"
Sep 16 17:34:53.664077 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:34:53.664042    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-gzwdn" event={"ID":"a9dc8c86-0e70-401e-b73e-65e7d9768032","Type":"ContainerStarted","Data":"46ea692db8317e5726399d4a2b341257bceb8237616ad8ee6d80f9d024fe5248"}
Sep 16 17:34:53.858402 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:34:53.858373    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-cc8594966-x2w6w"]
Sep 16 17:34:53.952357 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:34:53.952310    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b6acb17_7da6_4c72_a34b_81c81dab2f0b.slice/crio-7ce725e1cae02cec0bf131f7031906386dc45f14f5b38fafec1ddccd76b38240 WatchSource:0}: Error finding container 7ce725e1cae02cec0bf131f7031906386dc45f14f5b38fafec1ddccd76b38240: Status 404 returned error can't find the container with id 7ce725e1cae02cec0bf131f7031906386dc45f14f5b38fafec1ddccd76b38240
Sep 16 17:34:54.672264 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:34:54.672220    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-cc8594966-x2w6w" event={"ID":"5b6acb17-7da6-4c72-a34b-81c81dab2f0b","Type":"ContainerStarted","Data":"7ce725e1cae02cec0bf131f7031906386dc45f14f5b38fafec1ddccd76b38240"}
Sep 16 17:34:57.691917 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:34:57.691776    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-cc8594966-x2w6w" event={"ID":"5b6acb17-7da6-4c72-a34b-81c81dab2f0b","Type":"ContainerStarted","Data":"cad2f8285f8a2f4b547bbbe140dcb6f1b7f0357e610e623b48c37a380f25b5e8"}
Sep 16 17:34:57.691917 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:34:57.691885    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-cc8594966-x2w6w"
Sep 16 17:34:57.693679 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:34:57.693641    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-gzwdn" event={"ID":"a9dc8c86-0e70-401e-b73e-65e7d9768032","Type":"ContainerStarted","Data":"fa348cb671cca51254d43ee1592f1439e01277ea3b2c5a174d1f1ceabd7ea737"}
Sep 16 17:34:57.693833 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:34:57.693804    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-64568bcd49-gzwdn"
Sep 16 17:34:57.711884 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:34:57.711809    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-cc8594966-x2w6w" podStartSLOduration=2.13366933 podStartE2EDuration="5.7117878s" totalImagesPullingTime="3.57811847s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:34:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:34:57.709772998 +0000 UTC m=+610.179771133" watchObservedRunningTime="2026-09-16 17:34:57.7117878 +0000 UTC m=+610.181785936"
Sep 16 17:34:57.729813 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:34:57.729758    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-64568bcd49-gzwdn" podStartSLOduration=1.439630146 podStartE2EDuration="5.729744643s" totalImagesPullingTime="4.290114497s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:34:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:34:57.728675064 +0000 UTC m=+610.198673201" watchObservedRunningTime="2026-09-16 17:34:57.729744643 +0000 UTC m=+610.199742778"
Sep 16 17:35:03.700737 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:35:03.700706    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-64568bcd49-gzwdn"
Sep 16 17:35:28.703432 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:35:28.703389    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-cc8594966-x2w6w"
Sep 16 17:35:37.341517 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:35:37.341474    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-cc8594966-x2w6w"]
Sep 16 17:35:37.342009 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:35:37.341688    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve/kserve-controller-manager-cc8594966-x2w6w" podUID="5b6acb17-7da6-4c72-a34b-81c81dab2f0b" containerName="manager" containerID="cri-o://cad2f8285f8a2f4b547bbbe140dcb6f1b7f0357e610e623b48c37a380f25b5e8" gracePeriod=10
Sep 16 17:35:37.380081 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:35:37.380044    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-cc8594966-hz8vc"]
Sep 16 17:35:37.382602 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:35:37.382584    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-cc8594966-hz8vc"
Sep 16 17:35:37.394935 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:35:37.394899    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-cc8594966-hz8vc"]
Sep 16 17:35:37.479478 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:35:37.479440    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sw29f\" (UniqueName: \"kubernetes.io/projected/243a73bb-a95e-4c0d-8fda-c2d3a1236b9e-kube-api-access-sw29f\") pod \"kserve-controller-manager-cc8594966-hz8vc\" (UID: \"243a73bb-a95e-4c0d-8fda-c2d3a1236b9e\") " pod="kserve/kserve-controller-manager-cc8594966-hz8vc"
Sep 16 17:35:37.479647 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:35:37.479499    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/243a73bb-a95e-4c0d-8fda-c2d3a1236b9e-cert\") pod \"kserve-controller-manager-cc8594966-hz8vc\" (UID: \"243a73bb-a95e-4c0d-8fda-c2d3a1236b9e\") " pod="kserve/kserve-controller-manager-cc8594966-hz8vc"
Sep 16 17:35:37.580000 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:35:37.579971    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/243a73bb-a95e-4c0d-8fda-c2d3a1236b9e-cert\") pod \"kserve-controller-manager-cc8594966-hz8vc\" (UID: \"243a73bb-a95e-4c0d-8fda-c2d3a1236b9e\") " pod="kserve/kserve-controller-manager-cc8594966-hz8vc"
Sep 16 17:35:37.580127 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:35:37.580099    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sw29f\" (UniqueName: \"kubernetes.io/projected/243a73bb-a95e-4c0d-8fda-c2d3a1236b9e-kube-api-access-sw29f\") pod \"kserve-controller-manager-cc8594966-hz8vc\" (UID: \"243a73bb-a95e-4c0d-8fda-c2d3a1236b9e\") " pod="kserve/kserve-controller-manager-cc8594966-hz8vc"
Sep 16 17:35:37.582493 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:35:37.582464    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/243a73bb-a95e-4c0d-8fda-c2d3a1236b9e-cert\") pod \"kserve-controller-manager-cc8594966-hz8vc\" (UID: \"243a73bb-a95e-4c0d-8fda-c2d3a1236b9e\") " pod="kserve/kserve-controller-manager-cc8594966-hz8vc"
Sep 16 17:35:37.588915 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:35:37.588882    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sw29f\" (UniqueName: \"kubernetes.io/projected/243a73bb-a95e-4c0d-8fda-c2d3a1236b9e-kube-api-access-sw29f\") pod \"kserve-controller-manager-cc8594966-hz8vc\" (UID: \"243a73bb-a95e-4c0d-8fda-c2d3a1236b9e\") " pod="kserve/kserve-controller-manager-cc8594966-hz8vc"
Sep 16 17:35:37.607855 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:35:37.607761    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-cc8594966-x2w6w"
Sep 16 17:35:37.681193 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:35:37.681155    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4n2jn\" (UniqueName: \"kubernetes.io/projected/5b6acb17-7da6-4c72-a34b-81c81dab2f0b-kube-api-access-4n2jn\") pod \"5b6acb17-7da6-4c72-a34b-81c81dab2f0b\" (UID: \"5b6acb17-7da6-4c72-a34b-81c81dab2f0b\") "
Sep 16 17:35:37.681357 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:35:37.681265    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5b6acb17-7da6-4c72-a34b-81c81dab2f0b-cert\") pod \"5b6acb17-7da6-4c72-a34b-81c81dab2f0b\" (UID: \"5b6acb17-7da6-4c72-a34b-81c81dab2f0b\") "
Sep 16 17:35:37.683540 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:35:37.683500    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b6acb17-7da6-4c72-a34b-81c81dab2f0b-cert" (OuterVolumeSpecName: "cert") pod "5b6acb17-7da6-4c72-a34b-81c81dab2f0b" (UID: "5b6acb17-7da6-4c72-a34b-81c81dab2f0b"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 17:35:37.683656 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:35:37.683517    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b6acb17-7da6-4c72-a34b-81c81dab2f0b-kube-api-access-4n2jn" (OuterVolumeSpecName: "kube-api-access-4n2jn") pod "5b6acb17-7da6-4c72-a34b-81c81dab2f0b" (UID: "5b6acb17-7da6-4c72-a34b-81c81dab2f0b"). InnerVolumeSpecName "kube-api-access-4n2jn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 17:35:37.735695 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:35:37.735650    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-cc8594966-hz8vc"
Sep 16 17:35:37.781781 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:35:37.781747    2579 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5b6acb17-7da6-4c72-a34b-81c81dab2f0b-cert\") on node \"ip-10-0-138-73.ec2.internal\" DevicePath \"\""
Sep 16 17:35:37.781781 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:35:37.781779    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4n2jn\" (UniqueName: \"kubernetes.io/projected/5b6acb17-7da6-4c72-a34b-81c81dab2f0b-kube-api-access-4n2jn\") on node \"ip-10-0-138-73.ec2.internal\" DevicePath \"\""
Sep 16 17:35:37.885642 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:35:37.885548    2579 generic.go:356] "Generic (PLEG): container finished" podID="5b6acb17-7da6-4c72-a34b-81c81dab2f0b" containerID="cad2f8285f8a2f4b547bbbe140dcb6f1b7f0357e610e623b48c37a380f25b5e8" exitCode=0
Sep 16 17:35:37.885642 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:35:37.885592    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-cc8594966-x2w6w"
Sep 16 17:35:37.885642 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:35:37.885602    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-cc8594966-x2w6w" event={"ID":"5b6acb17-7da6-4c72-a34b-81c81dab2f0b","Type":"ContainerDied","Data":"cad2f8285f8a2f4b547bbbe140dcb6f1b7f0357e610e623b48c37a380f25b5e8"}
Sep 16 17:35:37.885902 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:35:37.885652    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-cc8594966-x2w6w" event={"ID":"5b6acb17-7da6-4c72-a34b-81c81dab2f0b","Type":"ContainerDied","Data":"7ce725e1cae02cec0bf131f7031906386dc45f14f5b38fafec1ddccd76b38240"}
Sep 16 17:35:37.885902 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:35:37.885679    2579 scope.go:117] "RemoveContainer" containerID="cad2f8285f8a2f4b547bbbe140dcb6f1b7f0357e610e623b48c37a380f25b5e8"
Sep 16 17:35:37.894446 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:35:37.894424    2579 scope.go:117] "RemoveContainer" containerID="cad2f8285f8a2f4b547bbbe140dcb6f1b7f0357e610e623b48c37a380f25b5e8"
Sep 16 17:35:37.894794 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:35:37.894769    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cad2f8285f8a2f4b547bbbe140dcb6f1b7f0357e610e623b48c37a380f25b5e8\": container with ID starting with cad2f8285f8a2f4b547bbbe140dcb6f1b7f0357e610e623b48c37a380f25b5e8 not found: ID does not exist" containerID="cad2f8285f8a2f4b547bbbe140dcb6f1b7f0357e610e623b48c37a380f25b5e8"
Sep 16 17:35:37.894898 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:35:37.894808    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cad2f8285f8a2f4b547bbbe140dcb6f1b7f0357e610e623b48c37a380f25b5e8"} err="failed to get container status \"cad2f8285f8a2f4b547bbbe140dcb6f1b7f0357e610e623b48c37a380f25b5e8\": rpc error: code = NotFound desc = could not find container \"cad2f8285f8a2f4b547bbbe140dcb6f1b7f0357e610e623b48c37a380f25b5e8\": container with ID starting with cad2f8285f8a2f4b547bbbe140dcb6f1b7f0357e610e623b48c37a380f25b5e8 not found: ID does not exist"
Sep 16 17:35:37.909320 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:35:37.909279    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-cc8594966-x2w6w"]
Sep 16 17:35:37.912339 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:35:37.912310    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/kserve-controller-manager-cc8594966-x2w6w"]
Sep 16 17:35:38.010575 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:35:38.010540    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-cc8594966-hz8vc"]
Sep 16 17:35:38.013793 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:35:38.013771    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod243a73bb_a95e_4c0d_8fda_c2d3a1236b9e.slice/crio-bcd2d772183d5bc2067936b0e1572760162fe2aeb6540a11a4bf8db2d67ab4bd WatchSource:0}: Error finding container bcd2d772183d5bc2067936b0e1572760162fe2aeb6540a11a4bf8db2d67ab4bd: Status 404 returned error can't find the container with id bcd2d772183d5bc2067936b0e1572760162fe2aeb6540a11a4bf8db2d67ab4bd
Sep 16 17:35:38.099572 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:35:38.099531    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5b6acb17-7da6-4c72-a34b-81c81dab2f0b" path="/var/lib/kubelet/pods/5b6acb17-7da6-4c72-a34b-81c81dab2f0b/volumes"
Sep 16 17:35:38.893359 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:35:38.893324    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-cc8594966-hz8vc" event={"ID":"243a73bb-a95e-4c0d-8fda-c2d3a1236b9e","Type":"ContainerStarted","Data":"9ba7a43dfeb2dd0f91a8788f88566d154bbeb2900ebc9283a87023c14bcaec22"}
Sep 16 17:35:38.893780 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:35:38.893374    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-cc8594966-hz8vc" event={"ID":"243a73bb-a95e-4c0d-8fda-c2d3a1236b9e","Type":"ContainerStarted","Data":"bcd2d772183d5bc2067936b0e1572760162fe2aeb6540a11a4bf8db2d67ab4bd"}
Sep 16 17:35:38.893780 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:35:38.893510    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-cc8594966-hz8vc"
Sep 16 17:35:38.912413 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:35:38.912366    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-cc8594966-hz8vc" podStartSLOduration=1.508612511 podStartE2EDuration="1.912349198s" totalImagesPullingTime="403.736687ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:35:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:35:38.911705422 +0000 UTC m=+651.381703557" watchObservedRunningTime="2026-09-16 17:35:38.912349198 +0000 UTC m=+651.382347332"
Sep 16 17:36:09.904180 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:36:09.904148    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-cc8594966-hz8vc"
Sep 16 17:36:11.463750 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:36:11.463716    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-7b954ffc95-dcwxf"]
Sep 16 17:36:11.464141 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:36:11.464112    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b6acb17-7da6-4c72-a34b-81c81dab2f0b" containerName="manager"
Sep 16 17:36:11.464141 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:36:11.464124    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b6acb17-7da6-4c72-a34b-81c81dab2f0b" containerName="manager"
Sep 16 17:36:11.464216 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:36:11.464191    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="5b6acb17-7da6-4c72-a34b-81c81dab2f0b" containerName="manager"
Sep 16 17:36:11.466431 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:36:11.466414    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-dcwxf"
Sep 16 17:36:11.471359 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:36:11.471333    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\""
Sep 16 17:36:11.471472 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:36:11.471437    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-kchlq\""
Sep 16 17:36:11.479869 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:36:11.479833    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-7b966d7984-wc25q"]
Sep 16 17:36:11.482244 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:36:11.482226    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-7b966d7984-wc25q"
Sep 16 17:36:11.485154 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:36:11.485135    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-metrics-tls\""
Sep 16 17:36:11.486453 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:36:11.486438    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-mxwn5\""
Sep 16 17:36:11.493086 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:36:11.493067    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\""
Sep 16 17:36:11.503321 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:36:11.503298    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-dcwxf"]
Sep 16 17:36:11.509681 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:36:11.509658    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-7b966d7984-wc25q"]
Sep 16 17:36:11.583895 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:36:11.583838    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a6ddf5b-ecfa-4bf7-9acc-a2e96c17676c-metrics-certs\") pod \"odh-model-controller-7b966d7984-wc25q\" (UID: \"8a6ddf5b-ecfa-4bf7-9acc-a2e96c17676c\") " pod="kserve/odh-model-controller-7b966d7984-wc25q"
Sep 16 17:36:11.583895 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:36:11.583898    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mlfn5\" (UniqueName: \"kubernetes.io/projected/f07f0722-dbc1-441a-8d3a-5b560f1d5e12-kube-api-access-mlfn5\") pod \"model-serving-api-7b954ffc95-dcwxf\" (UID: \"f07f0722-dbc1-441a-8d3a-5b560f1d5e12\") " pod="kserve/model-serving-api-7b954ffc95-dcwxf"
Sep 16 17:36:11.584154 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:36:11.583994    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f07f0722-dbc1-441a-8d3a-5b560f1d5e12-tls-certs\") pod \"model-serving-api-7b954ffc95-dcwxf\" (UID: \"f07f0722-dbc1-441a-8d3a-5b560f1d5e12\") " pod="kserve/model-serving-api-7b954ffc95-dcwxf"
Sep 16 17:36:11.584154 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:36:11.584029    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8a6ddf5b-ecfa-4bf7-9acc-a2e96c17676c-cert\") pod \"odh-model-controller-7b966d7984-wc25q\" (UID: \"8a6ddf5b-ecfa-4bf7-9acc-a2e96c17676c\") " pod="kserve/odh-model-controller-7b966d7984-wc25q"
Sep 16 17:36:11.584154 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:36:11.584100    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pprdl\" (UniqueName: \"kubernetes.io/projected/8a6ddf5b-ecfa-4bf7-9acc-a2e96c17676c-kube-api-access-pprdl\") pod \"odh-model-controller-7b966d7984-wc25q\" (UID: \"8a6ddf5b-ecfa-4bf7-9acc-a2e96c17676c\") " pod="kserve/odh-model-controller-7b966d7984-wc25q"
Sep 16 17:36:11.685139 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:36:11.685105    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a6ddf5b-ecfa-4bf7-9acc-a2e96c17676c-metrics-certs\") pod \"odh-model-controller-7b966d7984-wc25q\" (UID: \"8a6ddf5b-ecfa-4bf7-9acc-a2e96c17676c\") " pod="kserve/odh-model-controller-7b966d7984-wc25q"
Sep 16 17:36:11.685139 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:36:11.685146    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlfn5\" (UniqueName: \"kubernetes.io/projected/f07f0722-dbc1-441a-8d3a-5b560f1d5e12-kube-api-access-mlfn5\") pod \"model-serving-api-7b954ffc95-dcwxf\" (UID: \"f07f0722-dbc1-441a-8d3a-5b560f1d5e12\") " pod="kserve/model-serving-api-7b954ffc95-dcwxf"
Sep 16 17:36:11.685389 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:36:11.685169    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f07f0722-dbc1-441a-8d3a-5b560f1d5e12-tls-certs\") pod \"model-serving-api-7b954ffc95-dcwxf\" (UID: \"f07f0722-dbc1-441a-8d3a-5b560f1d5e12\") " pod="kserve/model-serving-api-7b954ffc95-dcwxf"
Sep 16 17:36:11.685389 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:36:11.685189    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8a6ddf5b-ecfa-4bf7-9acc-a2e96c17676c-cert\") pod \"odh-model-controller-7b966d7984-wc25q\" (UID: \"8a6ddf5b-ecfa-4bf7-9acc-a2e96c17676c\") " pod="kserve/odh-model-controller-7b966d7984-wc25q"
Sep 16 17:36:11.685389 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:36:11.685219    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pprdl\" (UniqueName: \"kubernetes.io/projected/8a6ddf5b-ecfa-4bf7-9acc-a2e96c17676c-kube-api-access-pprdl\") pod \"odh-model-controller-7b966d7984-wc25q\" (UID: \"8a6ddf5b-ecfa-4bf7-9acc-a2e96c17676c\") " pod="kserve/odh-model-controller-7b966d7984-wc25q"
Sep 16 17:36:11.685389 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:36:11.685262    2579 secret.go:189] Couldn't get secret kserve/model-serving-api-tls: secret "model-serving-api-tls" not found
Sep 16 17:36:11.685389 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:36:11.685321    2579 secret.go:189] Couldn't get secret kserve/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found
Sep 16 17:36:11.685389 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:36:11.685341    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f07f0722-dbc1-441a-8d3a-5b560f1d5e12-tls-certs podName:f07f0722-dbc1-441a-8d3a-5b560f1d5e12 nodeName:}" failed. No retries permitted until 2026-09-16 17:36:12.185321888 +0000 UTC m=+684.655320013 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/f07f0722-dbc1-441a-8d3a-5b560f1d5e12-tls-certs") pod "model-serving-api-7b954ffc95-dcwxf" (UID: "f07f0722-dbc1-441a-8d3a-5b560f1d5e12") : secret "model-serving-api-tls" not found
Sep 16 17:36:11.685389 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:36:11.685376    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a6ddf5b-ecfa-4bf7-9acc-a2e96c17676c-cert podName:8a6ddf5b-ecfa-4bf7-9acc-a2e96c17676c nodeName:}" failed. No retries permitted until 2026-09-16 17:36:12.185363142 +0000 UTC m=+684.655361256 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/8a6ddf5b-ecfa-4bf7-9acc-a2e96c17676c-cert") pod "odh-model-controller-7b966d7984-wc25q" (UID: "8a6ddf5b-ecfa-4bf7-9acc-a2e96c17676c") : secret "odh-model-controller-webhook-cert" not found
Sep 16 17:36:11.685733 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:36:11.685441    2579 secret.go:189] Couldn't get secret kserve/odh-model-controller-metrics-tls: secret "odh-model-controller-metrics-tls" not found
Sep 16 17:36:11.685733 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:36:11.685509    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a6ddf5b-ecfa-4bf7-9acc-a2e96c17676c-metrics-certs podName:8a6ddf5b-ecfa-4bf7-9acc-a2e96c17676c nodeName:}" failed. No retries permitted until 2026-09-16 17:36:12.185492472 +0000 UTC m=+684.655490588 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8a6ddf5b-ecfa-4bf7-9acc-a2e96c17676c-metrics-certs") pod "odh-model-controller-7b966d7984-wc25q" (UID: "8a6ddf5b-ecfa-4bf7-9acc-a2e96c17676c") : secret "odh-model-controller-metrics-tls" not found
Sep 16 17:36:11.710249 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:36:11.710214    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pprdl\" (UniqueName: \"kubernetes.io/projected/8a6ddf5b-ecfa-4bf7-9acc-a2e96c17676c-kube-api-access-pprdl\") pod \"odh-model-controller-7b966d7984-wc25q\" (UID: \"8a6ddf5b-ecfa-4bf7-9acc-a2e96c17676c\") " pod="kserve/odh-model-controller-7b966d7984-wc25q"
Sep 16 17:36:11.711997 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:36:11.711974    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mlfn5\" (UniqueName: \"kubernetes.io/projected/f07f0722-dbc1-441a-8d3a-5b560f1d5e12-kube-api-access-mlfn5\") pod \"model-serving-api-7b954ffc95-dcwxf\" (UID: \"f07f0722-dbc1-441a-8d3a-5b560f1d5e12\") " pod="kserve/model-serving-api-7b954ffc95-dcwxf"
Sep 16 17:36:12.189329 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:36:12.189294    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a6ddf5b-ecfa-4bf7-9acc-a2e96c17676c-metrics-certs\") pod \"odh-model-controller-7b966d7984-wc25q\" (UID: \"8a6ddf5b-ecfa-4bf7-9acc-a2e96c17676c\") " pod="kserve/odh-model-controller-7b966d7984-wc25q"
Sep 16 17:36:12.189543 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:36:12.189349    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f07f0722-dbc1-441a-8d3a-5b560f1d5e12-tls-certs\") pod \"model-serving-api-7b954ffc95-dcwxf\" (UID: \"f07f0722-dbc1-441a-8d3a-5b560f1d5e12\") " pod="kserve/model-serving-api-7b954ffc95-dcwxf"
Sep 16 17:36:12.189543 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:36:12.189380    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8a6ddf5b-ecfa-4bf7-9acc-a2e96c17676c-cert\") pod \"odh-model-controller-7b966d7984-wc25q\" (UID: \"8a6ddf5b-ecfa-4bf7-9acc-a2e96c17676c\") " pod="kserve/odh-model-controller-7b966d7984-wc25q"
Sep 16 17:36:12.189543 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:36:12.189459    2579 secret.go:189] Couldn't get secret kserve/odh-model-controller-metrics-tls: secret "odh-model-controller-metrics-tls" not found
Sep 16 17:36:12.189726 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:36:12.189555    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a6ddf5b-ecfa-4bf7-9acc-a2e96c17676c-metrics-certs podName:8a6ddf5b-ecfa-4bf7-9acc-a2e96c17676c nodeName:}" failed. No retries permitted until 2026-09-16 17:36:13.189537232 +0000 UTC m=+685.659535349 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8a6ddf5b-ecfa-4bf7-9acc-a2e96c17676c-metrics-certs") pod "odh-model-controller-7b966d7984-wc25q" (UID: "8a6ddf5b-ecfa-4bf7-9acc-a2e96c17676c") : secret "odh-model-controller-metrics-tls" not found
Sep 16 17:36:12.191820 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:36:12.191800    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f07f0722-dbc1-441a-8d3a-5b560f1d5e12-tls-certs\") pod \"model-serving-api-7b954ffc95-dcwxf\" (UID: \"f07f0722-dbc1-441a-8d3a-5b560f1d5e12\") " pod="kserve/model-serving-api-7b954ffc95-dcwxf"
Sep 16 17:36:12.191916 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:36:12.191830    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8a6ddf5b-ecfa-4bf7-9acc-a2e96c17676c-cert\") pod \"odh-model-controller-7b966d7984-wc25q\" (UID: \"8a6ddf5b-ecfa-4bf7-9acc-a2e96c17676c\") " pod="kserve/odh-model-controller-7b966d7984-wc25q"
Sep 16 17:36:12.376757 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:36:12.376720    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-dcwxf"
Sep 16 17:36:12.661173 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:36:12.661022    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-dcwxf"]
Sep 16 17:36:12.663786 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:36:12.663757    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf07f0722_dbc1_441a_8d3a_5b560f1d5e12.slice/crio-77b23d9878cd1b903fcfe18f33664ddc170fd2f90b8563d7d41235dc3c6550cd WatchSource:0}: Error finding container 77b23d9878cd1b903fcfe18f33664ddc170fd2f90b8563d7d41235dc3c6550cd: Status 404 returned error can't find the container with id 77b23d9878cd1b903fcfe18f33664ddc170fd2f90b8563d7d41235dc3c6550cd
Sep 16 17:36:13.051739 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:36:13.051702    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-dcwxf" event={"ID":"f07f0722-dbc1-441a-8d3a-5b560f1d5e12","Type":"ContainerStarted","Data":"77b23d9878cd1b903fcfe18f33664ddc170fd2f90b8563d7d41235dc3c6550cd"}
Sep 16 17:36:13.198341 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:36:13.198305    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a6ddf5b-ecfa-4bf7-9acc-a2e96c17676c-metrics-certs\") pod \"odh-model-controller-7b966d7984-wc25q\" (UID: \"8a6ddf5b-ecfa-4bf7-9acc-a2e96c17676c\") " pod="kserve/odh-model-controller-7b966d7984-wc25q"
Sep 16 17:36:13.201286 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:36:13.201251    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a6ddf5b-ecfa-4bf7-9acc-a2e96c17676c-metrics-certs\") pod \"odh-model-controller-7b966d7984-wc25q\" (UID: \"8a6ddf5b-ecfa-4bf7-9acc-a2e96c17676c\") " pod="kserve/odh-model-controller-7b966d7984-wc25q"
Sep 16 17:36:13.291855 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:36:13.291803    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-7b966d7984-wc25q"
Sep 16 17:36:13.602656 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:36:13.602578    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-7b966d7984-wc25q"]
Sep 16 17:36:13.607245 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:36:13.607214    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a6ddf5b_ecfa_4bf7_9acc_a2e96c17676c.slice/crio-f840089dcf8a1e1642236178958f10ba34b7e7f4039b854a0730d2e2610d8949 WatchSource:0}: Error finding container f840089dcf8a1e1642236178958f10ba34b7e7f4039b854a0730d2e2610d8949: Status 404 returned error can't find the container with id f840089dcf8a1e1642236178958f10ba34b7e7f4039b854a0730d2e2610d8949
Sep 16 17:36:14.064375 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:36:14.064336    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-7b966d7984-wc25q" event={"ID":"8a6ddf5b-ecfa-4bf7-9acc-a2e96c17676c","Type":"ContainerStarted","Data":"f840089dcf8a1e1642236178958f10ba34b7e7f4039b854a0730d2e2610d8949"}
Sep 16 17:36:15.074796 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:36:15.074696    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-dcwxf" event={"ID":"f07f0722-dbc1-441a-8d3a-5b560f1d5e12","Type":"ContainerStarted","Data":"aeefcbe995480577688dfcf28bdaceaab62490d1c9ccaac40f03c50465157e02"}
Sep 16 17:36:15.074796 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:36:15.074786    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-7b954ffc95-dcwxf"
Sep 16 17:36:15.096176 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:36:15.096107    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/model-serving-api-7b954ffc95-dcwxf" podStartSLOduration=1.9725302070000001 podStartE2EDuration="4.096086543s" totalImagesPullingTime="2.123556336s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:36:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:36:15.095808971 +0000 UTC m=+687.565807131" watchObservedRunningTime="2026-09-16 17:36:15.096086543 +0000 UTC m=+687.566084703"
Sep 16 17:36:18.092301 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:36:18.092268    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-7b966d7984-wc25q" event={"ID":"8a6ddf5b-ecfa-4bf7-9acc-a2e96c17676c","Type":"ContainerStarted","Data":"649a6e213ebca04d2772b7500fc96a26d16fb1813ad0e705712b0aa015df7397"}
Sep 16 17:36:18.092702 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:36:18.092389    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-7b966d7984-wc25q"
Sep 16 17:36:18.125878 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:36:18.125804    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/odh-model-controller-7b966d7984-wc25q" podStartSLOduration=3.547283249 podStartE2EDuration="7.125784551s" totalImagesPullingTime="3.578501302s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:36:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:36:18.123999923 +0000 UTC m=+690.593998080" watchObservedRunningTime="2026-09-16 17:36:18.125784551 +0000 UTC m=+690.595782687"
Sep 16 17:36:26.084619 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:36:26.084587    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-7b954ffc95-dcwxf"
Sep 16 17:36:29.099988 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:36:29.099949    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-7b966d7984-wc25q"
Sep 16 17:38:01.716012 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:38:01.715977    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-87956-predictor-6989556c8-zhw88"]
Sep 16 17:38:01.720185 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:38:01.720164    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-87956-predictor-6989556c8-zhw88"
Sep 16 17:38:01.726015 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:38:01.725989    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 17:38:01.726015 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:38:01.726013    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 16 17:38:01.726204 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:38:01.726185    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 16 17:38:01.727129 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:38:01.727096    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-9bc9t\""
Sep 16 17:38:01.727236 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:38:01.727210    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-raw-sklearn-batcher-87956-kube-rbac-proxy-sar-config\""
Sep 16 17:38:01.727312 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:38:01.727242    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 16 17:38:01.727312 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:38:01.727275    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-raw-sklearn-batcher-87956-predictor-serving-cert\""
Sep 16 17:38:01.743746 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:38:01.743716    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-87956-predictor-6989556c8-zhw88"]
Sep 16 17:38:01.841509 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:38:01.841463    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5ef287cf-ba59-471f-8512-919070a41924-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-87956-predictor-6989556c8-zhw88\" (UID: \"5ef287cf-ba59-471f-8512-919070a41924\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-87956-predictor-6989556c8-zhw88"
Sep 16 17:38:01.841696 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:38:01.841524    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5ef287cf-ba59-471f-8512-919070a41924-proxy-tls\") pod \"isvc-raw-sklearn-batcher-87956-predictor-6989556c8-zhw88\" (UID: \"5ef287cf-ba59-471f-8512-919070a41924\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-87956-predictor-6989556c8-zhw88"
Sep 16 17:38:01.841696 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:38:01.841572    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5ef287cf-ba59-471f-8512-919070a41924-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-87956-predictor-6989556c8-zhw88\" (UID: \"5ef287cf-ba59-471f-8512-919070a41924\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-87956-predictor-6989556c8-zhw88"
Sep 16 17:38:01.841696 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:38:01.841598    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5ef287cf-ba59-471f-8512-919070a41924-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-87956-predictor-6989556c8-zhw88\" (UID: \"5ef287cf-ba59-471f-8512-919070a41924\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-87956-predictor-6989556c8-zhw88"
Sep 16 17:38:01.841696 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:38:01.841656    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txd8j\" (UniqueName: \"kubernetes.io/projected/5ef287cf-ba59-471f-8512-919070a41924-kube-api-access-txd8j\") pod \"isvc-raw-sklearn-batcher-87956-predictor-6989556c8-zhw88\" (UID: \"5ef287cf-ba59-471f-8512-919070a41924\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-87956-predictor-6989556c8-zhw88"
Sep 16 17:38:01.942758 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:38:01.942715    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5ef287cf-ba59-471f-8512-919070a41924-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-87956-predictor-6989556c8-zhw88\" (UID: \"5ef287cf-ba59-471f-8512-919070a41924\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-87956-predictor-6989556c8-zhw88"
Sep 16 17:38:01.942991 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:38:01.942774    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5ef287cf-ba59-471f-8512-919070a41924-proxy-tls\") pod \"isvc-raw-sklearn-batcher-87956-predictor-6989556c8-zhw88\" (UID: \"5ef287cf-ba59-471f-8512-919070a41924\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-87956-predictor-6989556c8-zhw88"
Sep 16 17:38:01.942991 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:38:01.942817    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5ef287cf-ba59-471f-8512-919070a41924-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-87956-predictor-6989556c8-zhw88\" (UID: \"5ef287cf-ba59-471f-8512-919070a41924\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-87956-predictor-6989556c8-zhw88"
Sep 16 17:38:01.942991 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:38:01.942873    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5ef287cf-ba59-471f-8512-919070a41924-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-87956-predictor-6989556c8-zhw88\" (UID: \"5ef287cf-ba59-471f-8512-919070a41924\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-87956-predictor-6989556c8-zhw88"
Sep 16 17:38:01.942991 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:38:01.942931    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txd8j\" (UniqueName: \"kubernetes.io/projected/5ef287cf-ba59-471f-8512-919070a41924-kube-api-access-txd8j\") pod \"isvc-raw-sklearn-batcher-87956-predictor-6989556c8-zhw88\" (UID: \"5ef287cf-ba59-471f-8512-919070a41924\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-87956-predictor-6989556c8-zhw88"
Sep 16 17:38:01.943390 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:38:01.943366    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5ef287cf-ba59-471f-8512-919070a41924-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-87956-predictor-6989556c8-zhw88\" (UID: \"5ef287cf-ba59-471f-8512-919070a41924\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-87956-predictor-6989556c8-zhw88"
Sep 16 17:38:01.943570 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:38:01.943553    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5ef287cf-ba59-471f-8512-919070a41924-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-87956-predictor-6989556c8-zhw88\" (UID: \"5ef287cf-ba59-471f-8512-919070a41924\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-87956-predictor-6989556c8-zhw88"
Sep 16 17:38:01.943611 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:38:01.943552    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5ef287cf-ba59-471f-8512-919070a41924-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-87956-predictor-6989556c8-zhw88\" (UID: \"5ef287cf-ba59-471f-8512-919070a41924\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-87956-predictor-6989556c8-zhw88"
Sep 16 17:38:01.945308 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:38:01.945290    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5ef287cf-ba59-471f-8512-919070a41924-proxy-tls\") pod \"isvc-raw-sklearn-batcher-87956-predictor-6989556c8-zhw88\" (UID: \"5ef287cf-ba59-471f-8512-919070a41924\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-87956-predictor-6989556c8-zhw88"
Sep 16 17:38:01.953389 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:38:01.953357    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-txd8j\" (UniqueName: \"kubernetes.io/projected/5ef287cf-ba59-471f-8512-919070a41924-kube-api-access-txd8j\") pod \"isvc-raw-sklearn-batcher-87956-predictor-6989556c8-zhw88\" (UID: \"5ef287cf-ba59-471f-8512-919070a41924\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-87956-predictor-6989556c8-zhw88"
Sep 16 17:38:02.030421 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:38:02.030387    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-87956-predictor-6989556c8-zhw88"
Sep 16 17:38:02.310749 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:38:02.310724    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-87956-predictor-6989556c8-zhw88"]
Sep 16 17:38:02.313403 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:38:02.313357    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ef287cf_ba59_471f_8512_919070a41924.slice/crio-e9ca11385775be724898ce60ae59656cd7714fb09f4c01d4a1b32eb1b750c61e WatchSource:0}: Error finding container e9ca11385775be724898ce60ae59656cd7714fb09f4c01d4a1b32eb1b750c61e: Status 404 returned error can't find the container with id e9ca11385775be724898ce60ae59656cd7714fb09f4c01d4a1b32eb1b750c61e
Sep 16 17:38:02.315466 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:38:02.315449    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 17:38:02.575779 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:38:02.575694    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-87956-predictor-6989556c8-zhw88" event={"ID":"5ef287cf-ba59-471f-8512-919070a41924","Type":"ContainerStarted","Data":"e9ca11385775be724898ce60ae59656cd7714fb09f4c01d4a1b32eb1b750c61e"}
Sep 16 17:38:06.602278 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:38:06.602234    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-87956-predictor-6989556c8-zhw88" event={"ID":"5ef287cf-ba59-471f-8512-919070a41924","Type":"ContainerStarted","Data":"a90aa2885c8b0592dd6626bf0bb8df12f16a7bbbeca6d0d70444fbf173811de8"}
Sep 16 17:38:09.622772 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:38:09.622738    2579 generic.go:356] "Generic (PLEG): container finished" podID="5ef287cf-ba59-471f-8512-919070a41924" containerID="a90aa2885c8b0592dd6626bf0bb8df12f16a7bbbeca6d0d70444fbf173811de8" exitCode=0
Sep 16 17:38:09.623150 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:38:09.622812    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-87956-predictor-6989556c8-zhw88" event={"ID":"5ef287cf-ba59-471f-8512-919070a41924","Type":"ContainerDied","Data":"a90aa2885c8b0592dd6626bf0bb8df12f16a7bbbeca6d0d70444fbf173811de8"}
Sep 16 17:38:22.704025 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:38:22.703990    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-87956-predictor-6989556c8-zhw88" event={"ID":"5ef287cf-ba59-471f-8512-919070a41924","Type":"ContainerStarted","Data":"48edff71923cd6e3fddb0eb7260dcc8c3b441c736782d5d26283fa80df6263fc"}
Sep 16 17:38:25.723926 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:38:25.723884    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-87956-predictor-6989556c8-zhw88" event={"ID":"5ef287cf-ba59-471f-8512-919070a41924","Type":"ContainerStarted","Data":"4ee193c5f0a0689da0755d69d6f0c2adbc6d666faa7a850a37cdef1262b15517"}
Sep 16 17:38:27.737110 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:38:27.737072    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-87956-predictor-6989556c8-zhw88" event={"ID":"5ef287cf-ba59-471f-8512-919070a41924","Type":"ContainerStarted","Data":"f0c66e5bd87bfc0408e6a424e128774c62e24b9e9875729be696ab8f83395995"}
Sep 16 17:38:27.737598 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:38:27.737284    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-87956-predictor-6989556c8-zhw88"
Sep 16 17:38:27.737598 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:38:27.737430    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-87956-predictor-6989556c8-zhw88"
Sep 16 17:38:27.738894 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:38:27.738869    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-87956-predictor-6989556c8-zhw88" podUID="5ef287cf-ba59-471f-8512-919070a41924" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Sep 16 17:38:27.767576 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:38:27.767519    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-87956-predictor-6989556c8-zhw88" podStartSLOduration=2.6559716460000002 podStartE2EDuration="26.767504012s" totalImagesPullingTime="20.377411688s" totalInitContainerRuntime="3.734120678s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:38:01 +0000 UTC" imagePullSessionsCount=4 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:38:27.765045872 +0000 UTC m=+820.235044007" watchObservedRunningTime="2026-09-16 17:38:27.767504012 +0000 UTC m=+820.237502147"
Sep 16 17:38:28.741610 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:38:28.741574    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-87956-predictor-6989556c8-zhw88"
Sep 16 17:38:28.742019 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:38:28.741669    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-87956-predictor-6989556c8-zhw88" podUID="5ef287cf-ba59-471f-8512-919070a41924" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Sep 16 17:38:28.742751 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:38:28.742728    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-87956-predictor-6989556c8-zhw88" podUID="5ef287cf-ba59-471f-8512-919070a41924" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 17:38:29.746578 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:38:29.746536    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-87956-predictor-6989556c8-zhw88" podUID="5ef287cf-ba59-471f-8512-919070a41924" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Sep 16 17:38:29.747013 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:38:29.746922    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-87956-predictor-6989556c8-zhw88" podUID="5ef287cf-ba59-471f-8512-919070a41924" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 17:38:29.751757 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:38:29.751734    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-87956-predictor-6989556c8-zhw88"
Sep 16 17:38:30.756952 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:38:30.751315    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-87956-predictor-6989556c8-zhw88" podUID="5ef287cf-ba59-471f-8512-919070a41924" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Sep 16 17:38:30.757724 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:38:30.757688    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-87956-predictor-6989556c8-zhw88" podUID="5ef287cf-ba59-471f-8512-919070a41924" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 17:38:40.751239 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:38:40.751189    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-87956-predictor-6989556c8-zhw88" podUID="5ef287cf-ba59-471f-8512-919070a41924" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Sep 16 17:38:40.751729 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:38:40.751601    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-87956-predictor-6989556c8-zhw88" podUID="5ef287cf-ba59-471f-8512-919070a41924" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 17:38:50.751417 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:38:50.751365    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-87956-predictor-6989556c8-zhw88" podUID="5ef287cf-ba59-471f-8512-919070a41924" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Sep 16 17:38:50.751907 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:38:50.751873    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-87956-predictor-6989556c8-zhw88" podUID="5ef287cf-ba59-471f-8512-919070a41924" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 17:39:00.751163 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:39:00.751110    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-87956-predictor-6989556c8-zhw88" podUID="5ef287cf-ba59-471f-8512-919070a41924" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Sep 16 17:39:00.751641 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:39:00.751589    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-87956-predictor-6989556c8-zhw88" podUID="5ef287cf-ba59-471f-8512-919070a41924" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 17:39:10.751045 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:39:10.750994    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-87956-predictor-6989556c8-zhw88" podUID="5ef287cf-ba59-471f-8512-919070a41924" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Sep 16 17:39:10.751508 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:39:10.751366    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-87956-predictor-6989556c8-zhw88" podUID="5ef287cf-ba59-471f-8512-919070a41924" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 17:39:20.751563 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:39:20.751510    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-87956-predictor-6989556c8-zhw88" podUID="5ef287cf-ba59-471f-8512-919070a41924" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Sep 16 17:39:20.752096 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:39:20.751992    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-87956-predictor-6989556c8-zhw88" podUID="5ef287cf-ba59-471f-8512-919070a41924" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 17:39:30.751985 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:39:30.751937    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-87956-predictor-6989556c8-zhw88" podUID="5ef287cf-ba59-471f-8512-919070a41924" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Sep 16 17:39:30.752411 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:39:30.752351    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-87956-predictor-6989556c8-zhw88" podUID="5ef287cf-ba59-471f-8512-919070a41924" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 17:39:40.751701 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:39:40.751615    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-87956-predictor-6989556c8-zhw88"
Sep 16 17:39:40.752225 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:39:40.752163    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-87956-predictor-6989556c8-zhw88"
Sep 16 17:39:55.973231 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:39:55.973195    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-87956-predictor-6989556c8-zhw88"]
Sep 16 17:39:55.973780 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:39:55.973546    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-87956-predictor-6989556c8-zhw88" podUID="5ef287cf-ba59-471f-8512-919070a41924" containerName="kserve-container" containerID="cri-o://48edff71923cd6e3fddb0eb7260dcc8c3b441c736782d5d26283fa80df6263fc" gracePeriod=30
Sep 16 17:39:55.973780 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:39:55.973569    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-87956-predictor-6989556c8-zhw88" podUID="5ef287cf-ba59-471f-8512-919070a41924" containerName="agent" containerID="cri-o://f0c66e5bd87bfc0408e6a424e128774c62e24b9e9875729be696ab8f83395995" gracePeriod=30
Sep 16 17:39:55.973780 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:39:55.973570    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-87956-predictor-6989556c8-zhw88" podUID="5ef287cf-ba59-471f-8512-919070a41924" containerName="kube-rbac-proxy" containerID="cri-o://4ee193c5f0a0689da0755d69d6f0c2adbc6d666faa7a850a37cdef1262b15517" gracePeriod=30
Sep 16 17:39:56.154758 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:39:56.154719    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-7ca88-predictor-5bbddb5fcd-v6fd6"]
Sep 16 17:39:56.157907 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:39:56.157888    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-7ca88-predictor-5bbddb5fcd-v6fd6"
Sep 16 17:39:56.160585 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:39:56.160562    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-7ca88-kube-rbac-proxy-sar-config\""
Sep 16 17:39:56.160872 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:39:56.160821    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-7ca88-predictor-serving-cert\""
Sep 16 17:39:56.166573 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:39:56.166546    2579 generic.go:356] "Generic (PLEG): container finished" podID="5ef287cf-ba59-471f-8512-919070a41924" containerID="4ee193c5f0a0689da0755d69d6f0c2adbc6d666faa7a850a37cdef1262b15517" exitCode=2
Sep 16 17:39:56.166684 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:39:56.166590    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-87956-predictor-6989556c8-zhw88" event={"ID":"5ef287cf-ba59-471f-8512-919070a41924","Type":"ContainerDied","Data":"4ee193c5f0a0689da0755d69d6f0c2adbc6d666faa7a850a37cdef1262b15517"}
Sep 16 17:39:56.176115 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:39:56.176089    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-7ca88-predictor-5bbddb5fcd-v6fd6"]
Sep 16 17:39:56.241986 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:39:56.241913    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rzlf\" (UniqueName: \"kubernetes.io/projected/390f6c47-fef4-4d54-b616-e5988767fb0d-kube-api-access-7rzlf\") pod \"isvc-sklearn-graph-raw-7ca88-predictor-5bbddb5fcd-v6fd6\" (UID: \"390f6c47-fef4-4d54-b616-e5988767fb0d\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-7ca88-predictor-5bbddb5fcd-v6fd6"
Sep 16 17:39:56.241986 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:39:56.241955    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/390f6c47-fef4-4d54-b616-e5988767fb0d-proxy-tls\") pod \"isvc-sklearn-graph-raw-7ca88-predictor-5bbddb5fcd-v6fd6\" (UID: \"390f6c47-fef4-4d54-b616-e5988767fb0d\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-7ca88-predictor-5bbddb5fcd-v6fd6"
Sep 16 17:39:56.242207 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:39:56.241980    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/390f6c47-fef4-4d54-b616-e5988767fb0d-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-7ca88-predictor-5bbddb5fcd-v6fd6\" (UID: \"390f6c47-fef4-4d54-b616-e5988767fb0d\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-7ca88-predictor-5bbddb5fcd-v6fd6"
Sep 16 17:39:56.242207 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:39:56.242072    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/390f6c47-fef4-4d54-b616-e5988767fb0d-cabundle-cert\") pod \"isvc-sklearn-graph-raw-7ca88-predictor-5bbddb5fcd-v6fd6\" (UID: \"390f6c47-fef4-4d54-b616-e5988767fb0d\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-7ca88-predictor-5bbddb5fcd-v6fd6"
Sep 16 17:39:56.242207 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:39:56.242182    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/390f6c47-fef4-4d54-b616-e5988767fb0d-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-7ca88-predictor-5bbddb5fcd-v6fd6\" (UID: \"390f6c47-fef4-4d54-b616-e5988767fb0d\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-7ca88-predictor-5bbddb5fcd-v6fd6"
Sep 16 17:39:56.343087 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:39:56.343051    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/390f6c47-fef4-4d54-b616-e5988767fb0d-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-7ca88-predictor-5bbddb5fcd-v6fd6\" (UID: \"390f6c47-fef4-4d54-b616-e5988767fb0d\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-7ca88-predictor-5bbddb5fcd-v6fd6"
Sep 16 17:39:56.343276 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:39:56.343112    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rzlf\" (UniqueName: \"kubernetes.io/projected/390f6c47-fef4-4d54-b616-e5988767fb0d-kube-api-access-7rzlf\") pod \"isvc-sklearn-graph-raw-7ca88-predictor-5bbddb5fcd-v6fd6\" (UID: \"390f6c47-fef4-4d54-b616-e5988767fb0d\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-7ca88-predictor-5bbddb5fcd-v6fd6"
Sep 16 17:39:56.343276 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:39:56.343155    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/390f6c47-fef4-4d54-b616-e5988767fb0d-proxy-tls\") pod \"isvc-sklearn-graph-raw-7ca88-predictor-5bbddb5fcd-v6fd6\" (UID: \"390f6c47-fef4-4d54-b616-e5988767fb0d\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-7ca88-predictor-5bbddb5fcd-v6fd6"
Sep 16 17:39:56.343276 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:39:56.343189    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/390f6c47-fef4-4d54-b616-e5988767fb0d-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-7ca88-predictor-5bbddb5fcd-v6fd6\" (UID: \"390f6c47-fef4-4d54-b616-e5988767fb0d\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-7ca88-predictor-5bbddb5fcd-v6fd6"
Sep 16 17:39:56.343276 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:39:56.343224    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/390f6c47-fef4-4d54-b616-e5988767fb0d-cabundle-cert\") pod \"isvc-sklearn-graph-raw-7ca88-predictor-5bbddb5fcd-v6fd6\" (UID: \"390f6c47-fef4-4d54-b616-e5988767fb0d\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-7ca88-predictor-5bbddb5fcd-v6fd6"
Sep 16 17:39:56.343528 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:39:56.343504    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/390f6c47-fef4-4d54-b616-e5988767fb0d-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-7ca88-predictor-5bbddb5fcd-v6fd6\" (UID: \"390f6c47-fef4-4d54-b616-e5988767fb0d\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-7ca88-predictor-5bbddb5fcd-v6fd6"
Sep 16 17:39:56.343938 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:39:56.343913    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/390f6c47-fef4-4d54-b616-e5988767fb0d-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-7ca88-predictor-5bbddb5fcd-v6fd6\" (UID: \"390f6c47-fef4-4d54-b616-e5988767fb0d\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-7ca88-predictor-5bbddb5fcd-v6fd6"
Sep 16 17:39:56.344005 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:39:56.343932    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/390f6c47-fef4-4d54-b616-e5988767fb0d-cabundle-cert\") pod \"isvc-sklearn-graph-raw-7ca88-predictor-5bbddb5fcd-v6fd6\" (UID: \"390f6c47-fef4-4d54-b616-e5988767fb0d\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-7ca88-predictor-5bbddb5fcd-v6fd6"
Sep 16 17:39:56.345934 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:39:56.345904    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/390f6c47-fef4-4d54-b616-e5988767fb0d-proxy-tls\") pod \"isvc-sklearn-graph-raw-7ca88-predictor-5bbddb5fcd-v6fd6\" (UID: \"390f6c47-fef4-4d54-b616-e5988767fb0d\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-7ca88-predictor-5bbddb5fcd-v6fd6"
Sep 16 17:39:56.353608 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:39:56.353584    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rzlf\" (UniqueName: \"kubernetes.io/projected/390f6c47-fef4-4d54-b616-e5988767fb0d-kube-api-access-7rzlf\") pod \"isvc-sklearn-graph-raw-7ca88-predictor-5bbddb5fcd-v6fd6\" (UID: \"390f6c47-fef4-4d54-b616-e5988767fb0d\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-7ca88-predictor-5bbddb5fcd-v6fd6"
Sep 16 17:39:56.469580 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:39:56.469545    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-7ca88-predictor-5bbddb5fcd-v6fd6"
Sep 16 17:39:56.750561 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:39:56.750536    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-7ca88-predictor-5bbddb5fcd-v6fd6"]
Sep 16 17:39:56.752774 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:39:56.752746    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod390f6c47_fef4_4d54_b616_e5988767fb0d.slice/crio-b4ea817ffd2e80e3acc90bea470608a074417b99c71b24fadcdaceb84343fa02 WatchSource:0}: Error finding container b4ea817ffd2e80e3acc90bea470608a074417b99c71b24fadcdaceb84343fa02: Status 404 returned error can't find the container with id b4ea817ffd2e80e3acc90bea470608a074417b99c71b24fadcdaceb84343fa02
Sep 16 17:39:57.174164 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:39:57.174063    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-7ca88-predictor-5bbddb5fcd-v6fd6" event={"ID":"390f6c47-fef4-4d54-b616-e5988767fb0d","Type":"ContainerStarted","Data":"362d855b3def5f19450164cd2e8ce341a27e6bde7d3e6e27670c3883a0118415"}
Sep 16 17:39:57.174164 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:39:57.174135    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-7ca88-predictor-5bbddb5fcd-v6fd6" event={"ID":"390f6c47-fef4-4d54-b616-e5988767fb0d","Type":"ContainerStarted","Data":"b4ea817ffd2e80e3acc90bea470608a074417b99c71b24fadcdaceb84343fa02"}
Sep 16 17:39:59.746888 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:39:59.746823    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-87956-predictor-6989556c8-zhw88" podUID="5ef287cf-ba59-471f-8512-919070a41924" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.37:8643/healthz\": dial tcp 10.134.0.37:8643: connect: connection refused"
Sep 16 17:40:00.751452 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:40:00.751418    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-87956-predictor-6989556c8-zhw88" podUID="5ef287cf-ba59-471f-8512-919070a41924" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 17:40:00.752680 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:40:00.752657    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-87956-predictor-6989556c8-zhw88" podUID="5ef287cf-ba59-471f-8512-919070a41924" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Sep 16 17:40:01.196809 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:40:01.196775    2579 generic.go:356] "Generic (PLEG): container finished" podID="5ef287cf-ba59-471f-8512-919070a41924" containerID="48edff71923cd6e3fddb0eb7260dcc8c3b441c736782d5d26283fa80df6263fc" exitCode=0
Sep 16 17:40:01.197009 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:40:01.196868    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-87956-predictor-6989556c8-zhw88" event={"ID":"5ef287cf-ba59-471f-8512-919070a41924","Type":"ContainerDied","Data":"48edff71923cd6e3fddb0eb7260dcc8c3b441c736782d5d26283fa80df6263fc"}
Sep 16 17:40:01.198394 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:40:01.198373    2579 generic.go:356] "Generic (PLEG): container finished" podID="390f6c47-fef4-4d54-b616-e5988767fb0d" containerID="362d855b3def5f19450164cd2e8ce341a27e6bde7d3e6e27670c3883a0118415" exitCode=0
Sep 16 17:40:01.198525 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:40:01.198434    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-7ca88-predictor-5bbddb5fcd-v6fd6" event={"ID":"390f6c47-fef4-4d54-b616-e5988767fb0d","Type":"ContainerDied","Data":"362d855b3def5f19450164cd2e8ce341a27e6bde7d3e6e27670c3883a0118415"}
Sep 16 17:40:02.204751 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:40:02.204716    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-7ca88-predictor-5bbddb5fcd-v6fd6" event={"ID":"390f6c47-fef4-4d54-b616-e5988767fb0d","Type":"ContainerStarted","Data":"47dff3676fc13cd08690512cfcb35dd364526f410db59e37a4d38d645a20a254"}
Sep 16 17:40:02.205223 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:40:02.204780    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-7ca88-predictor-5bbddb5fcd-v6fd6" event={"ID":"390f6c47-fef4-4d54-b616-e5988767fb0d","Type":"ContainerStarted","Data":"b85e0593053859aabd1741976b1d592ef50c6b175ee63455bfc8a289781fa717"}
Sep 16 17:40:02.205223 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:40:02.205085    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-7ca88-predictor-5bbddb5fcd-v6fd6"
Sep 16 17:40:02.205328 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:40:02.205240    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-7ca88-predictor-5bbddb5fcd-v6fd6"
Sep 16 17:40:02.206636 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:40:02.206614    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-7ca88-predictor-5bbddb5fcd-v6fd6" podUID="390f6c47-fef4-4d54-b616-e5988767fb0d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 16 17:40:02.228421 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:40:02.228374    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-7ca88-predictor-5bbddb5fcd-v6fd6" podStartSLOduration=2.338651599 podStartE2EDuration="6.228359187s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.889707588s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:39:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:40:02.226835997 +0000 UTC m=+914.696834133" watchObservedRunningTime="2026-09-16 17:40:02.228359187 +0000 UTC m=+914.698357322"
Sep 16 17:40:03.213866 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:40:03.211778    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-7ca88-predictor-5bbddb5fcd-v6fd6" podUID="390f6c47-fef4-4d54-b616-e5988767fb0d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 16 17:40:04.747505 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:40:04.747452    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-87956-predictor-6989556c8-zhw88" podUID="5ef287cf-ba59-471f-8512-919070a41924" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.37:8643/healthz\": dial tcp 10.134.0.37:8643: connect: connection refused"
Sep 16 17:40:08.216081 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:40:08.216053    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-7ca88-predictor-5bbddb5fcd-v6fd6"
Sep 16 17:40:08.216737 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:40:08.216703    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-7ca88-predictor-5bbddb5fcd-v6fd6" podUID="390f6c47-fef4-4d54-b616-e5988767fb0d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 16 17:40:09.746707 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:40:09.746661    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-87956-predictor-6989556c8-zhw88" podUID="5ef287cf-ba59-471f-8512-919070a41924" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.37:8643/healthz\": dial tcp 10.134.0.37:8643: connect: connection refused"
Sep 16 17:40:09.747118 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:40:09.746820    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-87956-predictor-6989556c8-zhw88"
Sep 16 17:40:10.751667 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:40:10.751620    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-87956-predictor-6989556c8-zhw88" podUID="5ef287cf-ba59-471f-8512-919070a41924" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 17:40:10.752108 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:40:10.752086    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-87956-predictor-6989556c8-zhw88" podUID="5ef287cf-ba59-471f-8512-919070a41924" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Sep 16 17:40:14.747029 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:40:14.746981    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-87956-predictor-6989556c8-zhw88" podUID="5ef287cf-ba59-471f-8512-919070a41924" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.37:8643/healthz\": dial tcp 10.134.0.37:8643: connect: connection refused"
Sep 16 17:40:18.216683 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:40:18.216639    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-7ca88-predictor-5bbddb5fcd-v6fd6" podUID="390f6c47-fef4-4d54-b616-e5988767fb0d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 16 17:40:19.747468 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:40:19.747424    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-87956-predictor-6989556c8-zhw88" podUID="5ef287cf-ba59-471f-8512-919070a41924" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.37:8643/healthz\": dial tcp 10.134.0.37:8643: connect: connection refused"
Sep 16 17:40:20.751390 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:40:20.751346    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-87956-predictor-6989556c8-zhw88" podUID="5ef287cf-ba59-471f-8512-919070a41924" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 17:40:20.751766 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:40:20.751482    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-87956-predictor-6989556c8-zhw88"
Sep 16 17:40:20.752600 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:40:20.752576    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-87956-predictor-6989556c8-zhw88" podUID="5ef287cf-ba59-471f-8512-919070a41924" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Sep 16 17:40:20.752694 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:40:20.752682    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-87956-predictor-6989556c8-zhw88"
Sep 16 17:40:24.746755 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:40:24.746713    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-87956-predictor-6989556c8-zhw88" podUID="5ef287cf-ba59-471f-8512-919070a41924" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.37:8643/healthz\": dial tcp 10.134.0.37:8643: connect: connection refused"
Sep 16 17:40:26.122029 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:40:26.122000    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-87956-predictor-6989556c8-zhw88"
Sep 16 17:40:26.215228 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:40:26.215193    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5ef287cf-ba59-471f-8512-919070a41924-proxy-tls\") pod \"5ef287cf-ba59-471f-8512-919070a41924\" (UID: \"5ef287cf-ba59-471f-8512-919070a41924\") "
Sep 16 17:40:26.215409 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:40:26.215253    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5ef287cf-ba59-471f-8512-919070a41924-kserve-provision-location\") pod \"5ef287cf-ba59-471f-8512-919070a41924\" (UID: \"5ef287cf-ba59-471f-8512-919070a41924\") "
Sep 16 17:40:26.215409 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:40:26.215274    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5ef287cf-ba59-471f-8512-919070a41924-cabundle-cert\") pod \"5ef287cf-ba59-471f-8512-919070a41924\" (UID: \"5ef287cf-ba59-471f-8512-919070a41924\") "
Sep 16 17:40:26.215409 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:40:26.215304    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5ef287cf-ba59-471f-8512-919070a41924-kube-rbac-proxy-sar-config\") pod \"5ef287cf-ba59-471f-8512-919070a41924\" (UID: \"5ef287cf-ba59-471f-8512-919070a41924\") "
Sep 16 17:40:26.215409 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:40:26.215335    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-txd8j\" (UniqueName: \"kubernetes.io/projected/5ef287cf-ba59-471f-8512-919070a41924-kube-api-access-txd8j\") pod \"5ef287cf-ba59-471f-8512-919070a41924\" (UID: \"5ef287cf-ba59-471f-8512-919070a41924\") "
Sep 16 17:40:26.215642 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:40:26.215553    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ef287cf-ba59-471f-8512-919070a41924-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "5ef287cf-ba59-471f-8512-919070a41924" (UID: "5ef287cf-ba59-471f-8512-919070a41924"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:40:26.215704 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:40:26.215672    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ef287cf-ba59-471f-8512-919070a41924-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "5ef287cf-ba59-471f-8512-919070a41924" (UID: "5ef287cf-ba59-471f-8512-919070a41924"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 17:40:26.215821 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:40:26.215699    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ef287cf-ba59-471f-8512-919070a41924-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "5ef287cf-ba59-471f-8512-919070a41924" (UID: "5ef287cf-ba59-471f-8512-919070a41924"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 17:40:26.216504 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:40:26.215954    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5ef287cf-ba59-471f-8512-919070a41924-kserve-provision-location\") on node \"ip-10-0-138-73.ec2.internal\" DevicePath \"\""
Sep 16 17:40:26.216504 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:40:26.216304    2579 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5ef287cf-ba59-471f-8512-919070a41924-cabundle-cert\") on node \"ip-10-0-138-73.ec2.internal\" DevicePath \"\""
Sep 16 17:40:26.216504 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:40:26.216469    2579 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5ef287cf-ba59-471f-8512-919070a41924-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-73.ec2.internal\" DevicePath \"\""
Sep 16 17:40:26.217784 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:40:26.217755    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ef287cf-ba59-471f-8512-919070a41924-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "5ef287cf-ba59-471f-8512-919070a41924" (UID: "5ef287cf-ba59-471f-8512-919070a41924"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 17:40:26.217978 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:40:26.217959    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ef287cf-ba59-471f-8512-919070a41924-kube-api-access-txd8j" (OuterVolumeSpecName: "kube-api-access-txd8j") pod "5ef287cf-ba59-471f-8512-919070a41924" (UID: "5ef287cf-ba59-471f-8512-919070a41924"). InnerVolumeSpecName "kube-api-access-txd8j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 17:40:26.317006 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:40:26.316974    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5ef287cf-ba59-471f-8512-919070a41924-proxy-tls\") on node \"ip-10-0-138-73.ec2.internal\" DevicePath \"\""
Sep 16 17:40:26.317006 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:40:26.317002    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-txd8j\" (UniqueName: \"kubernetes.io/projected/5ef287cf-ba59-471f-8512-919070a41924-kube-api-access-txd8j\") on node \"ip-10-0-138-73.ec2.internal\" DevicePath \"\""
Sep 16 17:40:26.323802 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:40:26.323773    2579 generic.go:356] "Generic (PLEG): container finished" podID="5ef287cf-ba59-471f-8512-919070a41924" containerID="f0c66e5bd87bfc0408e6a424e128774c62e24b9e9875729be696ab8f83395995" exitCode=0
Sep 16 17:40:26.323964 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:40:26.323836    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-87956-predictor-6989556c8-zhw88"
Sep 16 17:40:26.323964 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:40:26.323866    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-87956-predictor-6989556c8-zhw88" event={"ID":"5ef287cf-ba59-471f-8512-919070a41924","Type":"ContainerDied","Data":"f0c66e5bd87bfc0408e6a424e128774c62e24b9e9875729be696ab8f83395995"}
Sep 16 17:40:26.323964 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:40:26.323952    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-87956-predictor-6989556c8-zhw88" event={"ID":"5ef287cf-ba59-471f-8512-919070a41924","Type":"ContainerDied","Data":"e9ca11385775be724898ce60ae59656cd7714fb09f4c01d4a1b32eb1b750c61e"}
Sep 16 17:40:26.324129 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:40:26.324002    2579 scope.go:117] "RemoveContainer" containerID="f0c66e5bd87bfc0408e6a424e128774c62e24b9e9875729be696ab8f83395995"
Sep 16 17:40:26.333028 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:40:26.333012    2579 scope.go:117] "RemoveContainer" containerID="4ee193c5f0a0689da0755d69d6f0c2adbc6d666faa7a850a37cdef1262b15517"
Sep 16 17:40:26.340567 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:40:26.340550    2579 scope.go:117] "RemoveContainer" containerID="48edff71923cd6e3fddb0eb7260dcc8c3b441c736782d5d26283fa80df6263fc"
Sep 16 17:40:26.347954 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:40:26.347936    2579 scope.go:117] "RemoveContainer" containerID="a90aa2885c8b0592dd6626bf0bb8df12f16a7bbbeca6d0d70444fbf173811de8"
Sep 16 17:40:26.353338 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:40:26.353296    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-87956-predictor-6989556c8-zhw88"]
Sep 16 17:40:26.355903 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:40:26.355877    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-87956-predictor-6989556c8-zhw88"]
Sep 16 17:40:26.356482 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:40:26.356469    2579 scope.go:117] "RemoveContainer" containerID="f0c66e5bd87bfc0408e6a424e128774c62e24b9e9875729be696ab8f83395995"
Sep 16 17:40:26.356812 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:40:26.356790    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0c66e5bd87bfc0408e6a424e128774c62e24b9e9875729be696ab8f83395995\": container with ID starting with f0c66e5bd87bfc0408e6a424e128774c62e24b9e9875729be696ab8f83395995 not found: ID does not exist" containerID="f0c66e5bd87bfc0408e6a424e128774c62e24b9e9875729be696ab8f83395995"
Sep 16 17:40:26.357000 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:40:26.356825    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0c66e5bd87bfc0408e6a424e128774c62e24b9e9875729be696ab8f83395995"} err="failed to get container status \"f0c66e5bd87bfc0408e6a424e128774c62e24b9e9875729be696ab8f83395995\": rpc error: code = NotFound desc = could not find container \"f0c66e5bd87bfc0408e6a424e128774c62e24b9e9875729be696ab8f83395995\": container with ID starting with f0c66e5bd87bfc0408e6a424e128774c62e24b9e9875729be696ab8f83395995 not found: ID does not exist"
Sep 16 17:40:26.357000 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:40:26.356864    2579 scope.go:117] "RemoveContainer" containerID="4ee193c5f0a0689da0755d69d6f0c2adbc6d666faa7a850a37cdef1262b15517"
Sep 16 17:40:26.357175 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:40:26.357157    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ee193c5f0a0689da0755d69d6f0c2adbc6d666faa7a850a37cdef1262b15517\": container with ID starting with 4ee193c5f0a0689da0755d69d6f0c2adbc6d666faa7a850a37cdef1262b15517 not found: ID does not exist" containerID="4ee193c5f0a0689da0755d69d6f0c2adbc6d666faa7a850a37cdef1262b15517"
Sep 16 17:40:26.357216 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:40:26.357181    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ee193c5f0a0689da0755d69d6f0c2adbc6d666faa7a850a37cdef1262b15517"} err="failed to get container status \"4ee193c5f0a0689da0755d69d6f0c2adbc6d666faa7a850a37cdef1262b15517\": rpc error: code = NotFound desc = could not find container \"4ee193c5f0a0689da0755d69d6f0c2adbc6d666faa7a850a37cdef1262b15517\": container with ID starting with 4ee193c5f0a0689da0755d69d6f0c2adbc6d666faa7a850a37cdef1262b15517 not found: ID does not exist"
Sep 16 17:40:26.357216 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:40:26.357201    2579 scope.go:117] "RemoveContainer" containerID="48edff71923cd6e3fddb0eb7260dcc8c3b441c736782d5d26283fa80df6263fc"
Sep 16 17:40:26.357467 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:40:26.357450    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48edff71923cd6e3fddb0eb7260dcc8c3b441c736782d5d26283fa80df6263fc\": container with ID starting with 48edff71923cd6e3fddb0eb7260dcc8c3b441c736782d5d26283fa80df6263fc not found: ID does not exist" containerID="48edff71923cd6e3fddb0eb7260dcc8c3b441c736782d5d26283fa80df6263fc"
Sep 16 17:40:26.357518 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:40:26.357474    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48edff71923cd6e3fddb0eb7260dcc8c3b441c736782d5d26283fa80df6263fc"} err="failed to get container status \"48edff71923cd6e3fddb0eb7260dcc8c3b441c736782d5d26283fa80df6263fc\": rpc error: code = NotFound desc = could not find container \"48edff71923cd6e3fddb0eb7260dcc8c3b441c736782d5d26283fa80df6263fc\": container with ID starting with 48edff71923cd6e3fddb0eb7260dcc8c3b441c736782d5d26283fa80df6263fc not found: ID does not exist"
Sep 16 17:40:26.357518 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:40:26.357487    2579 scope.go:117] "RemoveContainer" containerID="a90aa2885c8b0592dd6626bf0bb8df12f16a7bbbeca6d0d70444fbf173811de8"
Sep 16 17:40:26.357674 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:40:26.357660    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a90aa2885c8b0592dd6626bf0bb8df12f16a7bbbeca6d0d70444fbf173811de8\": container with ID starting with a90aa2885c8b0592dd6626bf0bb8df12f16a7bbbeca6d0d70444fbf173811de8 not found: ID does not exist" containerID="a90aa2885c8b0592dd6626bf0bb8df12f16a7bbbeca6d0d70444fbf173811de8"
Sep 16 17:40:26.357717 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:40:26.357676    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a90aa2885c8b0592dd6626bf0bb8df12f16a7bbbeca6d0d70444fbf173811de8"} err="failed to get container status \"a90aa2885c8b0592dd6626bf0bb8df12f16a7bbbeca6d0d70444fbf173811de8\": rpc error: code = NotFound desc = could not find container \"a90aa2885c8b0592dd6626bf0bb8df12f16a7bbbeca6d0d70444fbf173811de8\": container with ID starting with a90aa2885c8b0592dd6626bf0bb8df12f16a7bbbeca6d0d70444fbf173811de8 not found: ID does not exist"
Sep 16 17:40:28.099858 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:40:28.099815    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5ef287cf-ba59-471f-8512-919070a41924" path="/var/lib/kubelet/pods/5ef287cf-ba59-471f-8512-919070a41924/volumes"
Sep 16 17:40:28.216625 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:40:28.216588    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-7ca88-predictor-5bbddb5fcd-v6fd6" podUID="390f6c47-fef4-4d54-b616-e5988767fb0d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 16 17:40:38.216610 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:40:38.216572    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-7ca88-predictor-5bbddb5fcd-v6fd6" podUID="390f6c47-fef4-4d54-b616-e5988767fb0d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 16 17:40:48.216631 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:40:48.216589    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-7ca88-predictor-5bbddb5fcd-v6fd6" podUID="390f6c47-fef4-4d54-b616-e5988767fb0d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 16 17:40:58.216612 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:40:58.216569    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-7ca88-predictor-5bbddb5fcd-v6fd6" podUID="390f6c47-fef4-4d54-b616-e5988767fb0d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 16 17:41:08.222412 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:41:08.222348    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-7ca88-predictor-5bbddb5fcd-v6fd6" podUID="390f6c47-fef4-4d54-b616-e5988767fb0d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 16 17:41:09.094508 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:41:09.094472    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-7ca88-predictor-5bbddb5fcd-v6fd6" podUID="390f6c47-fef4-4d54-b616-e5988767fb0d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 16 17:41:19.095108 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:41:19.095027    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-7ca88-predictor-5bbddb5fcd-v6fd6"
Sep 16 17:41:26.194750 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:41:26.194717    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5bb8f495f4-g9h9r"]
Sep 16 17:41:26.195347 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:41:26.195256    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ef287cf-ba59-471f-8512-919070a41924" containerName="storage-initializer"
Sep 16 17:41:26.195347 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:41:26.195276    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ef287cf-ba59-471f-8512-919070a41924" containerName="storage-initializer"
Sep 16 17:41:26.195347 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:41:26.195292    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ef287cf-ba59-471f-8512-919070a41924" containerName="agent"
Sep 16 17:41:26.195347 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:41:26.195320    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ef287cf-ba59-471f-8512-919070a41924" containerName="agent"
Sep 16 17:41:26.195347 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:41:26.195333    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ef287cf-ba59-471f-8512-919070a41924" containerName="kserve-container"
Sep 16 17:41:26.195347 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:41:26.195343    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ef287cf-ba59-471f-8512-919070a41924" containerName="kserve-container"
Sep 16 17:41:26.195626 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:41:26.195364    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ef287cf-ba59-471f-8512-919070a41924" containerName="kube-rbac-proxy"
Sep 16 17:41:26.195626 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:41:26.195383    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ef287cf-ba59-471f-8512-919070a41924" containerName="kube-rbac-proxy"
Sep 16 17:41:26.195626 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:41:26.195473    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="5ef287cf-ba59-471f-8512-919070a41924" containerName="kube-rbac-proxy"
Sep 16 17:41:26.195626 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:41:26.195487    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="5ef287cf-ba59-471f-8512-919070a41924" containerName="agent"
Sep 16 17:41:26.195626 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:41:26.195500    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="5ef287cf-ba59-471f-8512-919070a41924" containerName="kserve-container"
Sep 16 17:41:26.199004 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:41:26.198981    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5bb8f495f4-g9h9r"
Sep 16 17:41:26.203278 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:41:26.203255    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-kube-rbac-proxy-sar-config\""
Sep 16 17:41:26.203411 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:41:26.203293    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-v2-predictor-serving-cert\""
Sep 16 17:41:26.210157 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:41:26.210132    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5bb8f495f4-g9h9r"]
Sep 16 17:41:26.291051 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:41:26.291015    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-5hmxg"]
Sep 16 17:41:26.295019 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:41:26.294998    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-5hmxg"
Sep 16 17:41:26.297597 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:41:26.297578    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-predictor-serving-cert\""
Sep 16 17:41:26.307192 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:41:26.307170    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-5hmxg"]
Sep 16 17:41:26.309995 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:41:26.309972    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b1bdce8d-38c5-4704-b368-f2f2bd274d38-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-5bb8f495f4-g9h9r\" (UID: \"b1bdce8d-38c5-4704-b368-f2f2bd274d38\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5bb8f495f4-g9h9r"
Sep 16 17:41:26.310108 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:41:26.310025    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b1bdce8d-38c5-4704-b368-f2f2bd274d38-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-5bb8f495f4-g9h9r\" (UID: \"b1bdce8d-38c5-4704-b368-f2f2bd274d38\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5bb8f495f4-g9h9r"
Sep 16 17:41:26.310176 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:41:26.310101    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8djj\" (UniqueName: \"kubernetes.io/projected/b1bdce8d-38c5-4704-b368-f2f2bd274d38-kube-api-access-q8djj\") pod \"isvc-canary-stop-v2-predictor-5bb8f495f4-g9h9r\" (UID: \"b1bdce8d-38c5-4704-b368-f2f2bd274d38\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5bb8f495f4-g9h9r"
Sep 16 17:41:26.310176 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:41:26.310133    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b1bdce8d-38c5-4704-b368-f2f2bd274d38-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-5bb8f495f4-g9h9r\" (UID: \"b1bdce8d-38c5-4704-b368-f2f2bd274d38\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5bb8f495f4-g9h9r"
Sep 16 17:41:26.310176 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:41:26.310157    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b1bdce8d-38c5-4704-b368-f2f2bd274d38-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-5bb8f495f4-g9h9r\" (UID: \"b1bdce8d-38c5-4704-b368-f2f2bd274d38\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5bb8f495f4-g9h9r"
Sep 16 17:41:26.410725 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:41:26.410695    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8djj\" (UniqueName: \"kubernetes.io/projected/b1bdce8d-38c5-4704-b368-f2f2bd274d38-kube-api-access-q8djj\") pod \"isvc-canary-stop-v2-predictor-5bb8f495f4-g9h9r\" (UID: \"b1bdce8d-38c5-4704-b368-f2f2bd274d38\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5bb8f495f4-g9h9r"
Sep 16 17:41:26.410725 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:41:26.410731    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/88e7bc7f-b5f0-472f-838d-9754899a9692-kserve-provision-location\") pod \"isvc-canary-stop-predictor-59fbfd5bc-5hmxg\" (UID: \"88e7bc7f-b5f0-472f-838d-9754899a9692\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-5hmxg"
Sep 16 17:41:26.411017 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:41:26.410754    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b1bdce8d-38c5-4704-b368-f2f2bd274d38-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-5bb8f495f4-g9h9r\" (UID: \"b1bdce8d-38c5-4704-b368-f2f2bd274d38\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5bb8f495f4-g9h9r"
Sep 16 17:41:26.411017 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:41:26.410778    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b1bdce8d-38c5-4704-b368-f2f2bd274d38-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-5bb8f495f4-g9h9r\" (UID: \"b1bdce8d-38c5-4704-b368-f2f2bd274d38\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5bb8f495f4-g9h9r"
Sep 16 17:41:26.411017 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:41:26.410839    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/88e7bc7f-b5f0-472f-838d-9754899a9692-proxy-tls\") pod \"isvc-canary-stop-predictor-59fbfd5bc-5hmxg\" (UID: \"88e7bc7f-b5f0-472f-838d-9754899a9692\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-5hmxg"
Sep 16 17:41:26.411017 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:41:26.410921    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b1bdce8d-38c5-4704-b368-f2f2bd274d38-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-5bb8f495f4-g9h9r\" (UID: \"b1bdce8d-38c5-4704-b368-f2f2bd274d38\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5bb8f495f4-g9h9r"
Sep 16 17:41:26.411017 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:41:26.410950    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/88e7bc7f-b5f0-472f-838d-9754899a9692-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-59fbfd5bc-5hmxg\" (UID: \"88e7bc7f-b5f0-472f-838d-9754899a9692\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-5hmxg"
Sep 16 17:41:26.411017 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:41:26.410976    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/88e7bc7f-b5f0-472f-838d-9754899a9692-cabundle-cert\") pod \"isvc-canary-stop-predictor-59fbfd5bc-5hmxg\" (UID: \"88e7bc7f-b5f0-472f-838d-9754899a9692\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-5hmxg"
Sep 16 17:41:26.411328 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:41:26.411042    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4xm9\" (UniqueName: \"kubernetes.io/projected/88e7bc7f-b5f0-472f-838d-9754899a9692-kube-api-access-m4xm9\") pod \"isvc-canary-stop-predictor-59fbfd5bc-5hmxg\" (UID: \"88e7bc7f-b5f0-472f-838d-9754899a9692\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-5hmxg"
Sep 16 17:41:26.411328 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:41:26.411081    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b1bdce8d-38c5-4704-b368-f2f2bd274d38-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-5bb8f495f4-g9h9r\" (UID: \"b1bdce8d-38c5-4704-b368-f2f2bd274d38\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5bb8f495f4-g9h9r"
Sep 16 17:41:26.411328 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:41:26.411274    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b1bdce8d-38c5-4704-b368-f2f2bd274d38-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-5bb8f495f4-g9h9r\" (UID: \"b1bdce8d-38c5-4704-b368-f2f2bd274d38\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5bb8f495f4-g9h9r"
Sep 16 17:41:26.411699 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:41:26.411675    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b1bdce8d-38c5-4704-b368-f2f2bd274d38-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-5bb8f495f4-g9h9r\" (UID: \"b1bdce8d-38c5-4704-b368-f2f2bd274d38\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5bb8f495f4-g9h9r"
Sep 16 17:41:26.411795 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:41:26.411713    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b1bdce8d-38c5-4704-b368-f2f2bd274d38-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-5bb8f495f4-g9h9r\" (UID: \"b1bdce8d-38c5-4704-b368-f2f2bd274d38\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5bb8f495f4-g9h9r"
Sep 16 17:41:26.413241 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:41:26.413217    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b1bdce8d-38c5-4704-b368-f2f2bd274d38-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-5bb8f495f4-g9h9r\" (UID: \"b1bdce8d-38c5-4704-b368-f2f2bd274d38\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5bb8f495f4-g9h9r"
Sep 16 17:41:26.418838 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:41:26.418813    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8djj\" (UniqueName: \"kubernetes.io/projected/b1bdce8d-38c5-4704-b368-f2f2bd274d38-kube-api-access-q8djj\") pod \"isvc-canary-stop-v2-predictor-5bb8f495f4-g9h9r\" (UID: \"b1bdce8d-38c5-4704-b368-f2f2bd274d38\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5bb8f495f4-g9h9r"
Sep 16 17:41:26.510017 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:41:26.509937    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5bb8f495f4-g9h9r"
Sep 16 17:41:26.512264 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:41:26.512242    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/88e7bc7f-b5f0-472f-838d-9754899a9692-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-59fbfd5bc-5hmxg\" (UID: \"88e7bc7f-b5f0-472f-838d-9754899a9692\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-5hmxg"
Sep 16 17:41:26.512324 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:41:26.512273    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/88e7bc7f-b5f0-472f-838d-9754899a9692-cabundle-cert\") pod \"isvc-canary-stop-predictor-59fbfd5bc-5hmxg\" (UID: \"88e7bc7f-b5f0-472f-838d-9754899a9692\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-5hmxg"
Sep 16 17:41:26.512324 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:41:26.512303    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4xm9\" (UniqueName: \"kubernetes.io/projected/88e7bc7f-b5f0-472f-838d-9754899a9692-kube-api-access-m4xm9\") pod \"isvc-canary-stop-predictor-59fbfd5bc-5hmxg\" (UID: \"88e7bc7f-b5f0-472f-838d-9754899a9692\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-5hmxg"
Sep 16 17:41:26.512407 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:41:26.512359    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/88e7bc7f-b5f0-472f-838d-9754899a9692-kserve-provision-location\") pod \"isvc-canary-stop-predictor-59fbfd5bc-5hmxg\" (UID: \"88e7bc7f-b5f0-472f-838d-9754899a9692\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-5hmxg"
Sep 16 17:41:26.512457 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:41:26.512421    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/88e7bc7f-b5f0-472f-838d-9754899a9692-proxy-tls\") pod \"isvc-canary-stop-predictor-59fbfd5bc-5hmxg\" (UID: \"88e7bc7f-b5f0-472f-838d-9754899a9692\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-5hmxg"
Sep 16 17:41:26.512806 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:41:26.512785    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/88e7bc7f-b5f0-472f-838d-9754899a9692-kserve-provision-location\") pod \"isvc-canary-stop-predictor-59fbfd5bc-5hmxg\" (UID: \"88e7bc7f-b5f0-472f-838d-9754899a9692\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-5hmxg"
Sep 16 17:41:26.513095 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:41:26.513071    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/88e7bc7f-b5f0-472f-838d-9754899a9692-cabundle-cert\") pod \"isvc-canary-stop-predictor-59fbfd5bc-5hmxg\" (UID: \"88e7bc7f-b5f0-472f-838d-9754899a9692\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-5hmxg"
Sep 16 17:41:26.513183 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:41:26.513093    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/88e7bc7f-b5f0-472f-838d-9754899a9692-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-59fbfd5bc-5hmxg\" (UID: \"88e7bc7f-b5f0-472f-838d-9754899a9692\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-5hmxg"
Sep 16 17:41:26.515034 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:41:26.515017    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/88e7bc7f-b5f0-472f-838d-9754899a9692-proxy-tls\") pod \"isvc-canary-stop-predictor-59fbfd5bc-5hmxg\" (UID: \"88e7bc7f-b5f0-472f-838d-9754899a9692\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-5hmxg"
Sep 16 17:41:26.520622 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:41:26.520601    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4xm9\" (UniqueName: \"kubernetes.io/projected/88e7bc7f-b5f0-472f-838d-9754899a9692-kube-api-access-m4xm9\") pod \"isvc-canary-stop-predictor-59fbfd5bc-5hmxg\" (UID: \"88e7bc7f-b5f0-472f-838d-9754899a9692\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-5hmxg"
Sep 16 17:41:26.605257 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:41:26.605221    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-5hmxg"
Sep 16 17:41:26.786881 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:41:26.786834    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5bb8f495f4-g9h9r"]
Sep 16 17:41:26.790298 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:41:26.790271    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1bdce8d_38c5_4704_b368_f2f2bd274d38.slice/crio-eced8f5b0376512b46c497bf499d15ae633f105f3d9d984685a45f6409dd79ea WatchSource:0}: Error finding container eced8f5b0376512b46c497bf499d15ae633f105f3d9d984685a45f6409dd79ea: Status 404 returned error can't find the container with id eced8f5b0376512b46c497bf499d15ae633f105f3d9d984685a45f6409dd79ea
Sep 16 17:41:26.883692 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:41:26.883664    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-5hmxg"]
Sep 16 17:41:26.884737 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:41:26.884714    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88e7bc7f_b5f0_472f_838d_9754899a9692.slice/crio-2c6271f30aca48bc402183d5e6c10d8782e9af570437e76e4aa58c4d93889a3d WatchSource:0}: Error finding container 2c6271f30aca48bc402183d5e6c10d8782e9af570437e76e4aa58c4d93889a3d: Status 404 returned error can't find the container with id 2c6271f30aca48bc402183d5e6c10d8782e9af570437e76e4aa58c4d93889a3d
Sep 16 17:41:27.619631 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:41:27.619591    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-5hmxg" event={"ID":"88e7bc7f-b5f0-472f-838d-9754899a9692","Type":"ContainerStarted","Data":"a0c0b8003574dfa33442f31f2e4a2301b6e5c81555dcd0ed4682c569307f8a77"}
Sep 16 17:41:27.620067 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:41:27.619697    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-5hmxg" event={"ID":"88e7bc7f-b5f0-472f-838d-9754899a9692","Type":"ContainerStarted","Data":"2c6271f30aca48bc402183d5e6c10d8782e9af570437e76e4aa58c4d93889a3d"}
Sep 16 17:41:27.621300 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:41:27.621276    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5bb8f495f4-g9h9r" event={"ID":"b1bdce8d-38c5-4704-b368-f2f2bd274d38","Type":"ContainerStarted","Data":"b32def8e4c7093d47c34d4a9f8886ec403cf4b42a1cb36e9782f2a991596a625"}
Sep 16 17:41:27.621421 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:41:27.621326    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5bb8f495f4-g9h9r" event={"ID":"b1bdce8d-38c5-4704-b368-f2f2bd274d38","Type":"ContainerStarted","Data":"eced8f5b0376512b46c497bf499d15ae633f105f3d9d984685a45f6409dd79ea"}
Sep 16 17:41:31.644909 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:41:31.644874    2579 generic.go:356] "Generic (PLEG): container finished" podID="88e7bc7f-b5f0-472f-838d-9754899a9692" containerID="a0c0b8003574dfa33442f31f2e4a2301b6e5c81555dcd0ed4682c569307f8a77" exitCode=0
Sep 16 17:41:31.645338 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:41:31.644949    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-5hmxg" event={"ID":"88e7bc7f-b5f0-472f-838d-9754899a9692","Type":"ContainerDied","Data":"a0c0b8003574dfa33442f31f2e4a2301b6e5c81555dcd0ed4682c569307f8a77"}
Sep 16 17:41:31.646572 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:41:31.646549    2579 generic.go:356] "Generic (PLEG): container finished" podID="b1bdce8d-38c5-4704-b368-f2f2bd274d38" containerID="b32def8e4c7093d47c34d4a9f8886ec403cf4b42a1cb36e9782f2a991596a625" exitCode=0
Sep 16 17:41:31.646655 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:41:31.646619    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5bb8f495f4-g9h9r" event={"ID":"b1bdce8d-38c5-4704-b368-f2f2bd274d38","Type":"ContainerDied","Data":"b32def8e4c7093d47c34d4a9f8886ec403cf4b42a1cb36e9782f2a991596a625"}
Sep 16 17:41:32.654181 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:41:32.654148    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-5hmxg" event={"ID":"88e7bc7f-b5f0-472f-838d-9754899a9692","Type":"ContainerStarted","Data":"8c79f301df0db735cbb950e62916023a122c6d8e1efdd78e5a5c9a9821239450"}
Sep 16 17:41:32.654632 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:41:32.654210    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-5hmxg" event={"ID":"88e7bc7f-b5f0-472f-838d-9754899a9692","Type":"ContainerStarted","Data":"5448b5cc02c6c23262a75f66c070fca2525cccbde3486a74c275761fcbd789e4"}
Sep 16 17:41:32.654632 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:41:32.654504    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-5hmxg"
Sep 16 17:41:32.654733 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:41:32.654653    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-5hmxg"
Sep 16 17:41:32.656283 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:41:32.656256    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-5hmxg" podUID="88e7bc7f-b5f0-472f-838d-9754899a9692" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Sep 16 17:41:32.656469 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:41:32.656447    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5bb8f495f4-g9h9r" event={"ID":"b1bdce8d-38c5-4704-b368-f2f2bd274d38","Type":"ContainerStarted","Data":"373aae2492a9a535a43881dd29712fb5c7ee5785ed681e569c929665c4c6d7e4"}
Sep 16 17:41:32.656525 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:41:32.656501    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5bb8f495f4-g9h9r" event={"ID":"b1bdce8d-38c5-4704-b368-f2f2bd274d38","Type":"ContainerStarted","Data":"0217bb35b73abb198db934c75b6e3d99843568049ee2fcb9f8a0c2b70038ce50"}
Sep 16 17:41:32.656699 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:41:32.656685    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5bb8f495f4-g9h9r"
Sep 16 17:41:32.656745 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:41:32.656722    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5bb8f495f4-g9h9r"
Sep 16 17:41:32.657810 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:41:32.657786    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5bb8f495f4-g9h9r" podUID="b1bdce8d-38c5-4704-b368-f2f2bd274d38" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Sep 16 17:41:32.681035 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:41:32.680985    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-5hmxg" podStartSLOduration=2.699032944 podStartE2EDuration="6.680969606s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.981936662s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:41:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:41:32.678292347 +0000 UTC m=+1005.148290485" watchObservedRunningTime="2026-09-16 17:41:32.680969606 +0000 UTC m=+1005.150967742"
Sep 16 17:41:32.703150 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:41:32.703105    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5bb8f495f4-g9h9r" podStartSLOduration=2.844930277 podStartE2EDuration="6.703089092s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.858158815s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:41:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:41:32.699313296 +0000 UTC m=+1005.169311431" watchObservedRunningTime="2026-09-16 17:41:32.703089092 +0000 UTC m=+1005.173087227"
Sep 16 17:41:33.661163 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:41:33.661117    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-5hmxg" podUID="88e7bc7f-b5f0-472f-838d-9754899a9692" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Sep 16 17:41:33.661619 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:41:33.661164    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5bb8f495f4-g9h9r" podUID="b1bdce8d-38c5-4704-b368-f2f2bd274d38" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Sep 16 17:41:38.667147 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:41:38.667114    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-5hmxg"
Sep 16 17:41:38.667616 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:41:38.667275    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5bb8f495f4-g9h9r"
Sep 16 17:41:38.667616 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:41:38.667485    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-5hmxg" podUID="88e7bc7f-b5f0-472f-838d-9754899a9692" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Sep 16 17:41:38.668093 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:41:38.668069    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5bb8f495f4-g9h9r" podUID="b1bdce8d-38c5-4704-b368-f2f2bd274d38" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Sep 16 17:41:46.286401 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:41:46.286366    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-7ca88-predictor-5bbddb5fcd-v6fd6"]
Sep 16 17:41:46.286975 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:41:46.286673    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-7ca88-predictor-5bbddb5fcd-v6fd6" podUID="390f6c47-fef4-4d54-b616-e5988767fb0d" containerName="kserve-container" containerID="cri-o://b85e0593053859aabd1741976b1d592ef50c6b175ee63455bfc8a289781fa717" gracePeriod=30
Sep 16 17:41:46.286975 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:41:46.286737    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-7ca88-predictor-5bbddb5fcd-v6fd6" podUID="390f6c47-fef4-4d54-b616-e5988767fb0d" containerName="kube-rbac-proxy" containerID="cri-o://47dff3676fc13cd08690512cfcb35dd364526f410db59e37a4d38d645a20a254" gracePeriod=30
Sep 16 17:41:46.726805 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:41:46.726722    2579 generic.go:356] "Generic (PLEG): container finished" podID="390f6c47-fef4-4d54-b616-e5988767fb0d" containerID="47dff3676fc13cd08690512cfcb35dd364526f410db59e37a4d38d645a20a254" exitCode=2
Sep 16 17:41:46.726805 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:41:46.726781    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-7ca88-predictor-5bbddb5fcd-v6fd6" event={"ID":"390f6c47-fef4-4d54-b616-e5988767fb0d","Type":"ContainerDied","Data":"47dff3676fc13cd08690512cfcb35dd364526f410db59e37a4d38d645a20a254"}
Sep 16 17:41:48.211863 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:41:48.211805    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-7ca88-predictor-5bbddb5fcd-v6fd6" podUID="390f6c47-fef4-4d54-b616-e5988767fb0d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.38:8643/healthz\": dial tcp 10.134.0.38:8643: connect: connection refused"
Sep 16 17:41:48.667671 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:41:48.667624    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-5hmxg" podUID="88e7bc7f-b5f0-472f-838d-9754899a9692" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Sep 16 17:41:48.668037 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:41:48.668009    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5bb8f495f4-g9h9r" podUID="b1bdce8d-38c5-4704-b368-f2f2bd274d38" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Sep 16 17:41:49.094816 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:41:49.094775    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-7ca88-predictor-5bbddb5fcd-v6fd6" podUID="390f6c47-fef4-4d54-b616-e5988767fb0d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 16 17:41:51.534702 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:41:51.534679    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-7ca88-predictor-5bbddb5fcd-v6fd6"
Sep 16 17:41:51.639742 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:41:51.639651    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7rzlf\" (UniqueName: \"kubernetes.io/projected/390f6c47-fef4-4d54-b616-e5988767fb0d-kube-api-access-7rzlf\") pod \"390f6c47-fef4-4d54-b616-e5988767fb0d\" (UID: \"390f6c47-fef4-4d54-b616-e5988767fb0d\") "
Sep 16 17:41:51.639742 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:41:51.639691    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/390f6c47-fef4-4d54-b616-e5988767fb0d-proxy-tls\") pod \"390f6c47-fef4-4d54-b616-e5988767fb0d\" (UID: \"390f6c47-fef4-4d54-b616-e5988767fb0d\") "
Sep 16 17:41:51.640008 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:41:51.639760    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/390f6c47-fef4-4d54-b616-e5988767fb0d-kserve-provision-location\") pod \"390f6c47-fef4-4d54-b616-e5988767fb0d\" (UID: \"390f6c47-fef4-4d54-b616-e5988767fb0d\") "
Sep 16 17:41:51.640008 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:41:51.639815    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/390f6c47-fef4-4d54-b616-e5988767fb0d-kube-rbac-proxy-sar-config\") pod \"390f6c47-fef4-4d54-b616-e5988767fb0d\" (UID: \"390f6c47-fef4-4d54-b616-e5988767fb0d\") "
Sep 16 17:41:51.640008 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:41:51.639867    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/390f6c47-fef4-4d54-b616-e5988767fb0d-cabundle-cert\") pod \"390f6c47-fef4-4d54-b616-e5988767fb0d\" (UID: \"390f6c47-fef4-4d54-b616-e5988767fb0d\") "
Sep 16 17:41:51.640237 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:41:51.640201    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/390f6c47-fef4-4d54-b616-e5988767fb0d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "390f6c47-fef4-4d54-b616-e5988767fb0d" (UID: "390f6c47-fef4-4d54-b616-e5988767fb0d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:41:51.640237 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:41:51.640219    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/390f6c47-fef4-4d54-b616-e5988767fb0d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "390f6c47-fef4-4d54-b616-e5988767fb0d" (UID: "390f6c47-fef4-4d54-b616-e5988767fb0d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 17:41:51.640426 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:41:51.640257    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/390f6c47-fef4-4d54-b616-e5988767fb0d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "390f6c47-fef4-4d54-b616-e5988767fb0d" (UID: "390f6c47-fef4-4d54-b616-e5988767fb0d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 17:41:51.641905 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:41:51.641879    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/390f6c47-fef4-4d54-b616-e5988767fb0d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "390f6c47-fef4-4d54-b616-e5988767fb0d" (UID: "390f6c47-fef4-4d54-b616-e5988767fb0d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 17:41:51.641905 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:41:51.641897    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/390f6c47-fef4-4d54-b616-e5988767fb0d-kube-api-access-7rzlf" (OuterVolumeSpecName: "kube-api-access-7rzlf") pod "390f6c47-fef4-4d54-b616-e5988767fb0d" (UID: "390f6c47-fef4-4d54-b616-e5988767fb0d"). InnerVolumeSpecName "kube-api-access-7rzlf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 17:41:51.741560 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:41:51.741524    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7rzlf\" (UniqueName: \"kubernetes.io/projected/390f6c47-fef4-4d54-b616-e5988767fb0d-kube-api-access-7rzlf\") on node \"ip-10-0-138-73.ec2.internal\" DevicePath \"\""
Sep 16 17:41:51.741560 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:41:51.741554    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/390f6c47-fef4-4d54-b616-e5988767fb0d-proxy-tls\") on node \"ip-10-0-138-73.ec2.internal\" DevicePath \"\""
Sep 16 17:41:51.741560 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:41:51.741565    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/390f6c47-fef4-4d54-b616-e5988767fb0d-kserve-provision-location\") on node \"ip-10-0-138-73.ec2.internal\" DevicePath \"\""
Sep 16 17:41:51.741801 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:41:51.741574    2579 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/390f6c47-fef4-4d54-b616-e5988767fb0d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-73.ec2.internal\" DevicePath \"\""
Sep 16 17:41:51.741801 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:41:51.741583    2579 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/390f6c47-fef4-4d54-b616-e5988767fb0d-cabundle-cert\") on node \"ip-10-0-138-73.ec2.internal\" DevicePath \"\""
Sep 16 17:41:51.753516 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:41:51.753486    2579 generic.go:356] "Generic (PLEG): container finished" podID="390f6c47-fef4-4d54-b616-e5988767fb0d" containerID="b85e0593053859aabd1741976b1d592ef50c6b175ee63455bfc8a289781fa717" exitCode=0
Sep 16 17:41:51.753658 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:41:51.753539    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-7ca88-predictor-5bbddb5fcd-v6fd6"
Sep 16 17:41:51.753658 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:41:51.753578    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-7ca88-predictor-5bbddb5fcd-v6fd6" event={"ID":"390f6c47-fef4-4d54-b616-e5988767fb0d","Type":"ContainerDied","Data":"b85e0593053859aabd1741976b1d592ef50c6b175ee63455bfc8a289781fa717"}
Sep 16 17:41:51.753658 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:41:51.753654    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-7ca88-predictor-5bbddb5fcd-v6fd6" event={"ID":"390f6c47-fef4-4d54-b616-e5988767fb0d","Type":"ContainerDied","Data":"b4ea817ffd2e80e3acc90bea470608a074417b99c71b24fadcdaceb84343fa02"}
Sep 16 17:41:51.753826 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:41:51.753695    2579 scope.go:117] "RemoveContainer" containerID="47dff3676fc13cd08690512cfcb35dd364526f410db59e37a4d38d645a20a254"
Sep 16 17:41:51.762792 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:41:51.762775    2579 scope.go:117] "RemoveContainer" containerID="b85e0593053859aabd1741976b1d592ef50c6b175ee63455bfc8a289781fa717"
Sep 16 17:41:51.771471 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:41:51.771450    2579 scope.go:117] "RemoveContainer" containerID="362d855b3def5f19450164cd2e8ce341a27e6bde7d3e6e27670c3883a0118415"
Sep 16 17:41:51.779336 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:41:51.779311    2579 scope.go:117] "RemoveContainer" containerID="47dff3676fc13cd08690512cfcb35dd364526f410db59e37a4d38d645a20a254"
Sep 16 17:41:51.779725 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:41:51.779701    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47dff3676fc13cd08690512cfcb35dd364526f410db59e37a4d38d645a20a254\": container with ID starting with 47dff3676fc13cd08690512cfcb35dd364526f410db59e37a4d38d645a20a254 not found: ID does not exist" containerID="47dff3676fc13cd08690512cfcb35dd364526f410db59e37a4d38d645a20a254"
Sep 16 17:41:51.779820 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:41:51.779741    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47dff3676fc13cd08690512cfcb35dd364526f410db59e37a4d38d645a20a254"} err="failed to get container status \"47dff3676fc13cd08690512cfcb35dd364526f410db59e37a4d38d645a20a254\": rpc error: code = NotFound desc = could not find container \"47dff3676fc13cd08690512cfcb35dd364526f410db59e37a4d38d645a20a254\": container with ID starting with 47dff3676fc13cd08690512cfcb35dd364526f410db59e37a4d38d645a20a254 not found: ID does not exist"
Sep 16 17:41:51.779820 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:41:51.779767    2579 scope.go:117] "RemoveContainer" containerID="b85e0593053859aabd1741976b1d592ef50c6b175ee63455bfc8a289781fa717"
Sep 16 17:41:51.780300 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:41:51.780276    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b85e0593053859aabd1741976b1d592ef50c6b175ee63455bfc8a289781fa717\": container with ID starting with b85e0593053859aabd1741976b1d592ef50c6b175ee63455bfc8a289781fa717 not found: ID does not exist" containerID="b85e0593053859aabd1741976b1d592ef50c6b175ee63455bfc8a289781fa717"
Sep 16 17:41:51.780381 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:41:51.780307    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b85e0593053859aabd1741976b1d592ef50c6b175ee63455bfc8a289781fa717"} err="failed to get container status \"b85e0593053859aabd1741976b1d592ef50c6b175ee63455bfc8a289781fa717\": rpc error: code = NotFound desc = could not find container \"b85e0593053859aabd1741976b1d592ef50c6b175ee63455bfc8a289781fa717\": container with ID starting with b85e0593053859aabd1741976b1d592ef50c6b175ee63455bfc8a289781fa717 not found: ID does not exist"
Sep 16 17:41:51.780381 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:41:51.780325    2579 scope.go:117] "RemoveContainer" containerID="362d855b3def5f19450164cd2e8ce341a27e6bde7d3e6e27670c3883a0118415"
Sep 16 17:41:51.780657 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:41:51.780640    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"362d855b3def5f19450164cd2e8ce341a27e6bde7d3e6e27670c3883a0118415\": container with ID starting with 362d855b3def5f19450164cd2e8ce341a27e6bde7d3e6e27670c3883a0118415 not found: ID does not exist" containerID="362d855b3def5f19450164cd2e8ce341a27e6bde7d3e6e27670c3883a0118415"
Sep 16 17:41:51.780708 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:41:51.780663    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"362d855b3def5f19450164cd2e8ce341a27e6bde7d3e6e27670c3883a0118415"} err="failed to get container status \"362d855b3def5f19450164cd2e8ce341a27e6bde7d3e6e27670c3883a0118415\": rpc error: code = NotFound desc = could not find container \"362d855b3def5f19450164cd2e8ce341a27e6bde7d3e6e27670c3883a0118415\": container with ID starting with 362d855b3def5f19450164cd2e8ce341a27e6bde7d3e6e27670c3883a0118415 not found: ID does not exist"
Sep 16 17:41:51.781834 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:41:51.781816    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-7ca88-predictor-5bbddb5fcd-v6fd6"]
Sep 16 17:41:51.790494 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:41:51.790472    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-7ca88-predictor-5bbddb5fcd-v6fd6"]
Sep 16 17:41:52.098899 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:41:52.098865    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="390f6c47-fef4-4d54-b616-e5988767fb0d" path="/var/lib/kubelet/pods/390f6c47-fef4-4d54-b616-e5988767fb0d/volumes"
Sep 16 17:41:58.668298 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:41:58.668256    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5bb8f495f4-g9h9r" podUID="b1bdce8d-38c5-4704-b368-f2f2bd274d38" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Sep 16 17:41:58.668777 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:41:58.668304    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-5hmxg" podUID="88e7bc7f-b5f0-472f-838d-9754899a9692" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Sep 16 17:42:08.668148 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:42:08.668104    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-5hmxg" podUID="88e7bc7f-b5f0-472f-838d-9754899a9692" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Sep 16 17:42:08.668559 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:42:08.668121    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5bb8f495f4-g9h9r" podUID="b1bdce8d-38c5-4704-b368-f2f2bd274d38" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Sep 16 17:42:18.667762 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:42:18.667723    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-5hmxg" podUID="88e7bc7f-b5f0-472f-838d-9754899a9692" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Sep 16 17:42:18.668306 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:42:18.668102    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5bb8f495f4-g9h9r" podUID="b1bdce8d-38c5-4704-b368-f2f2bd274d38" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Sep 16 17:42:28.668079 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:42:28.668029    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5bb8f495f4-g9h9r" podUID="b1bdce8d-38c5-4704-b368-f2f2bd274d38" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Sep 16 17:42:28.668079 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:42:28.668063    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-5hmxg" podUID="88e7bc7f-b5f0-472f-838d-9754899a9692" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Sep 16 17:42:38.667985 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:42:38.667944    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-5hmxg" podUID="88e7bc7f-b5f0-472f-838d-9754899a9692" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Sep 16 17:42:38.668459 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:42:38.668053    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5bb8f495f4-g9h9r" podUID="b1bdce8d-38c5-4704-b368-f2f2bd274d38" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Sep 16 17:42:41.094436 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:42:41.094349    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-5hmxg" podUID="88e7bc7f-b5f0-472f-838d-9754899a9692" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Sep 16 17:42:48.669283 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:42:48.669253    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5bb8f495f4-g9h9r"
Sep 16 17:42:51.095077 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:42:51.094980    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-5hmxg"
Sep 16 17:42:56.190501 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:42:56.190469    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5bb8f495f4-g9h9r"]
Sep 16 17:42:56.191079 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:42:56.190779    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5bb8f495f4-g9h9r" podUID="b1bdce8d-38c5-4704-b368-f2f2bd274d38" containerName="kserve-container" containerID="cri-o://0217bb35b73abb198db934c75b6e3d99843568049ee2fcb9f8a0c2b70038ce50" gracePeriod=30
Sep 16 17:42:56.191079 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:42:56.190803    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5bb8f495f4-g9h9r" podUID="b1bdce8d-38c5-4704-b368-f2f2bd274d38" containerName="kube-rbac-proxy" containerID="cri-o://373aae2492a9a535a43881dd29712fb5c7ee5785ed681e569c929665c4c6d7e4" gracePeriod=30
Sep 16 17:42:56.223102 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:42:56.223069    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-5hmxg"]
Sep 16 17:42:56.223301 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:42:56.223274    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-5hmxg" podUID="88e7bc7f-b5f0-472f-838d-9754899a9692" containerName="kserve-container" containerID="cri-o://5448b5cc02c6c23262a75f66c070fca2525cccbde3486a74c275761fcbd789e4" gracePeriod=30
Sep 16 17:42:56.223383 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:42:56.223309    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-5hmxg" podUID="88e7bc7f-b5f0-472f-838d-9754899a9692" containerName="kube-rbac-proxy" containerID="cri-o://8c79f301df0db735cbb950e62916023a122c6d8e1efdd78e5a5c9a9821239450" gracePeriod=30
Sep 16 17:42:57.081040 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:42:57.081007    2579 generic.go:356] "Generic (PLEG): container finished" podID="88e7bc7f-b5f0-472f-838d-9754899a9692" containerID="8c79f301df0db735cbb950e62916023a122c6d8e1efdd78e5a5c9a9821239450" exitCode=2
Sep 16 17:42:57.081224 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:42:57.081090    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-5hmxg" event={"ID":"88e7bc7f-b5f0-472f-838d-9754899a9692","Type":"ContainerDied","Data":"8c79f301df0db735cbb950e62916023a122c6d8e1efdd78e5a5c9a9821239450"}
Sep 16 17:42:57.083084 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:42:57.083062    2579 generic.go:356] "Generic (PLEG): container finished" podID="b1bdce8d-38c5-4704-b368-f2f2bd274d38" containerID="373aae2492a9a535a43881dd29712fb5c7ee5785ed681e569c929665c4c6d7e4" exitCode=2
Sep 16 17:42:57.083213 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:42:57.083130    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5bb8f495f4-g9h9r" event={"ID":"b1bdce8d-38c5-4704-b368-f2f2bd274d38","Type":"ContainerDied","Data":"373aae2492a9a535a43881dd29712fb5c7ee5785ed681e569c929665c4c6d7e4"}
Sep 16 17:42:58.662059 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:42:58.662021    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-5hmxg" podUID="88e7bc7f-b5f0-472f-838d-9754899a9692" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.40:8643/healthz\": dial tcp 10.134.0.40:8643: connect: connection refused"
Sep 16 17:42:58.662448 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:42:58.662024    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5bb8f495f4-g9h9r" podUID="b1bdce8d-38c5-4704-b368-f2f2bd274d38" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.39:8643/healthz\": dial tcp 10.134.0.39:8643: connect: connection refused"
Sep 16 17:42:58.668201 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:42:58.668169    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5bb8f495f4-g9h9r" podUID="b1bdce8d-38c5-4704-b368-f2f2bd274d38" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Sep 16 17:43:01.078489 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:43:01.078461    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-5hmxg"
Sep 16 17:43:01.108946 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:43:01.108908    2579 generic.go:356] "Generic (PLEG): container finished" podID="88e7bc7f-b5f0-472f-838d-9754899a9692" containerID="5448b5cc02c6c23262a75f66c070fca2525cccbde3486a74c275761fcbd789e4" exitCode=0
Sep 16 17:43:01.109119 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:43:01.108970    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-5hmxg"
Sep 16 17:43:01.109119 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:43:01.108987    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-5hmxg" event={"ID":"88e7bc7f-b5f0-472f-838d-9754899a9692","Type":"ContainerDied","Data":"5448b5cc02c6c23262a75f66c070fca2525cccbde3486a74c275761fcbd789e4"}
Sep 16 17:43:01.109119 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:43:01.109057    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-5hmxg" event={"ID":"88e7bc7f-b5f0-472f-838d-9754899a9692","Type":"ContainerDied","Data":"2c6271f30aca48bc402183d5e6c10d8782e9af570437e76e4aa58c4d93889a3d"}
Sep 16 17:43:01.109119 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:43:01.109088    2579 scope.go:117] "RemoveContainer" containerID="8c79f301df0db735cbb950e62916023a122c6d8e1efdd78e5a5c9a9821239450"
Sep 16 17:43:01.111701 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:43:01.111674    2579 generic.go:356] "Generic (PLEG): container finished" podID="b1bdce8d-38c5-4704-b368-f2f2bd274d38" containerID="0217bb35b73abb198db934c75b6e3d99843568049ee2fcb9f8a0c2b70038ce50" exitCode=0
Sep 16 17:43:01.111866 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:43:01.111750    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5bb8f495f4-g9h9r" event={"ID":"b1bdce8d-38c5-4704-b368-f2f2bd274d38","Type":"ContainerDied","Data":"0217bb35b73abb198db934c75b6e3d99843568049ee2fcb9f8a0c2b70038ce50"}
Sep 16 17:43:01.119512 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:43:01.119492    2579 scope.go:117] "RemoveContainer" containerID="5448b5cc02c6c23262a75f66c070fca2525cccbde3486a74c275761fcbd789e4"
Sep 16 17:43:01.136191 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:43:01.136172    2579 scope.go:117] "RemoveContainer" containerID="a0c0b8003574dfa33442f31f2e4a2301b6e5c81555dcd0ed4682c569307f8a77"
Sep 16 17:43:01.137861 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:43:01.137829    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/88e7bc7f-b5f0-472f-838d-9754899a9692-kserve-provision-location\") pod \"88e7bc7f-b5f0-472f-838d-9754899a9692\" (UID: \"88e7bc7f-b5f0-472f-838d-9754899a9692\") "
Sep 16 17:43:01.137967 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:43:01.137876    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/88e7bc7f-b5f0-472f-838d-9754899a9692-proxy-tls\") pod \"88e7bc7f-b5f0-472f-838d-9754899a9692\" (UID: \"88e7bc7f-b5f0-472f-838d-9754899a9692\") "
Sep 16 17:43:01.137967 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:43:01.137902    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/88e7bc7f-b5f0-472f-838d-9754899a9692-kube-rbac-proxy-sar-config\") pod \"88e7bc7f-b5f0-472f-838d-9754899a9692\" (UID: \"88e7bc7f-b5f0-472f-838d-9754899a9692\") "
Sep 16 17:43:01.137967 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:43:01.137941    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/88e7bc7f-b5f0-472f-838d-9754899a9692-cabundle-cert\") pod \"88e7bc7f-b5f0-472f-838d-9754899a9692\" (UID: \"88e7bc7f-b5f0-472f-838d-9754899a9692\") "
Sep 16 17:43:01.138125 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:43:01.137965    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m4xm9\" (UniqueName: \"kubernetes.io/projected/88e7bc7f-b5f0-472f-838d-9754899a9692-kube-api-access-m4xm9\") pod \"88e7bc7f-b5f0-472f-838d-9754899a9692\" (UID: \"88e7bc7f-b5f0-472f-838d-9754899a9692\") "
Sep 16 17:43:01.138515 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:43:01.138485    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88e7bc7f-b5f0-472f-838d-9754899a9692-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "88e7bc7f-b5f0-472f-838d-9754899a9692" (UID: "88e7bc7f-b5f0-472f-838d-9754899a9692"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:43:01.138624 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:43:01.138564    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88e7bc7f-b5f0-472f-838d-9754899a9692-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "88e7bc7f-b5f0-472f-838d-9754899a9692" (UID: "88e7bc7f-b5f0-472f-838d-9754899a9692"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 17:43:01.138624 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:43:01.138575    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88e7bc7f-b5f0-472f-838d-9754899a9692-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "88e7bc7f-b5f0-472f-838d-9754899a9692" (UID: "88e7bc7f-b5f0-472f-838d-9754899a9692"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 17:43:01.140790 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:43:01.140769    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88e7bc7f-b5f0-472f-838d-9754899a9692-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "88e7bc7f-b5f0-472f-838d-9754899a9692" (UID: "88e7bc7f-b5f0-472f-838d-9754899a9692"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 17:43:01.140790 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:43:01.140784    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88e7bc7f-b5f0-472f-838d-9754899a9692-kube-api-access-m4xm9" (OuterVolumeSpecName: "kube-api-access-m4xm9") pod "88e7bc7f-b5f0-472f-838d-9754899a9692" (UID: "88e7bc7f-b5f0-472f-838d-9754899a9692"). InnerVolumeSpecName "kube-api-access-m4xm9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 17:43:01.141234 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:43:01.141216    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5bb8f495f4-g9h9r"
Sep 16 17:43:01.144791 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:43:01.144767    2579 scope.go:117] "RemoveContainer" containerID="8c79f301df0db735cbb950e62916023a122c6d8e1efdd78e5a5c9a9821239450"
Sep 16 17:43:01.145112 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:43:01.145093    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c79f301df0db735cbb950e62916023a122c6d8e1efdd78e5a5c9a9821239450\": container with ID starting with 8c79f301df0db735cbb950e62916023a122c6d8e1efdd78e5a5c9a9821239450 not found: ID does not exist" containerID="8c79f301df0db735cbb950e62916023a122c6d8e1efdd78e5a5c9a9821239450"
Sep 16 17:43:01.145191 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:43:01.145120    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c79f301df0db735cbb950e62916023a122c6d8e1efdd78e5a5c9a9821239450"} err="failed to get container status \"8c79f301df0db735cbb950e62916023a122c6d8e1efdd78e5a5c9a9821239450\": rpc error: code = NotFound desc = could not find container \"8c79f301df0db735cbb950e62916023a122c6d8e1efdd78e5a5c9a9821239450\": container with ID starting with 8c79f301df0db735cbb950e62916023a122c6d8e1efdd78e5a5c9a9821239450 not found: ID does not exist"
Sep 16 17:43:01.145191 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:43:01.145137    2579 scope.go:117] "RemoveContainer" containerID="5448b5cc02c6c23262a75f66c070fca2525cccbde3486a74c275761fcbd789e4"
Sep 16 17:43:01.145411 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:43:01.145392    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5448b5cc02c6c23262a75f66c070fca2525cccbde3486a74c275761fcbd789e4\": container with ID starting with 5448b5cc02c6c23262a75f66c070fca2525cccbde3486a74c275761fcbd789e4 not found: ID does not exist" containerID="5448b5cc02c6c23262a75f66c070fca2525cccbde3486a74c275761fcbd789e4"
Sep 16 17:43:01.145470 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:43:01.145416    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5448b5cc02c6c23262a75f66c070fca2525cccbde3486a74c275761fcbd789e4"} err="failed to get container status \"5448b5cc02c6c23262a75f66c070fca2525cccbde3486a74c275761fcbd789e4\": rpc error: code = NotFound desc = could not find container \"5448b5cc02c6c23262a75f66c070fca2525cccbde3486a74c275761fcbd789e4\": container with ID starting with 5448b5cc02c6c23262a75f66c070fca2525cccbde3486a74c275761fcbd789e4 not found: ID does not exist"
Sep 16 17:43:01.145470 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:43:01.145428    2579 scope.go:117] "RemoveContainer" containerID="a0c0b8003574dfa33442f31f2e4a2301b6e5c81555dcd0ed4682c569307f8a77"
Sep 16 17:43:01.145720 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:43:01.145705    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0c0b8003574dfa33442f31f2e4a2301b6e5c81555dcd0ed4682c569307f8a77\": container with ID starting with a0c0b8003574dfa33442f31f2e4a2301b6e5c81555dcd0ed4682c569307f8a77 not found: ID does not exist" containerID="a0c0b8003574dfa33442f31f2e4a2301b6e5c81555dcd0ed4682c569307f8a77"
Sep 16 17:43:01.145765 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:43:01.145722    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0c0b8003574dfa33442f31f2e4a2301b6e5c81555dcd0ed4682c569307f8a77"} err="failed to get container status \"a0c0b8003574dfa33442f31f2e4a2301b6e5c81555dcd0ed4682c569307f8a77\": rpc error: code = NotFound desc = could not find container \"a0c0b8003574dfa33442f31f2e4a2301b6e5c81555dcd0ed4682c569307f8a77\": container with ID starting with a0c0b8003574dfa33442f31f2e4a2301b6e5c81555dcd0ed4682c569307f8a77 not found: ID does not exist"
Sep 16 17:43:01.238905 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:43:01.238869    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b1bdce8d-38c5-4704-b368-f2f2bd274d38-proxy-tls\") pod \"b1bdce8d-38c5-4704-b368-f2f2bd274d38\" (UID: \"b1bdce8d-38c5-4704-b368-f2f2bd274d38\") "
Sep 16 17:43:01.239087 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:43:01.238937    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b1bdce8d-38c5-4704-b368-f2f2bd274d38-kube-rbac-proxy-sar-config\") pod \"b1bdce8d-38c5-4704-b368-f2f2bd274d38\" (UID: \"b1bdce8d-38c5-4704-b368-f2f2bd274d38\") "
Sep 16 17:43:01.239087 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:43:01.238954    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b1bdce8d-38c5-4704-b368-f2f2bd274d38-cabundle-cert\") pod \"b1bdce8d-38c5-4704-b368-f2f2bd274d38\" (UID: \"b1bdce8d-38c5-4704-b368-f2f2bd274d38\") "
Sep 16 17:43:01.239087 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:43:01.239017    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q8djj\" (UniqueName: \"kubernetes.io/projected/b1bdce8d-38c5-4704-b368-f2f2bd274d38-kube-api-access-q8djj\") pod \"b1bdce8d-38c5-4704-b368-f2f2bd274d38\" (UID: \"b1bdce8d-38c5-4704-b368-f2f2bd274d38\") "
Sep 16 17:43:01.239087 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:43:01.239034    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b1bdce8d-38c5-4704-b368-f2f2bd274d38-kserve-provision-location\") pod \"b1bdce8d-38c5-4704-b368-f2f2bd274d38\" (UID: \"b1bdce8d-38c5-4704-b368-f2f2bd274d38\") "
Sep 16 17:43:01.239299 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:43:01.239233    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/88e7bc7f-b5f0-472f-838d-9754899a9692-kserve-provision-location\") on node \"ip-10-0-138-73.ec2.internal\" DevicePath \"\""
Sep 16 17:43:01.239299 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:43:01.239245    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/88e7bc7f-b5f0-472f-838d-9754899a9692-proxy-tls\") on node \"ip-10-0-138-73.ec2.internal\" DevicePath \"\""
Sep 16 17:43:01.239299 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:43:01.239258    2579 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/88e7bc7f-b5f0-472f-838d-9754899a9692-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-73.ec2.internal\" DevicePath \"\""
Sep 16 17:43:01.239299 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:43:01.239270    2579 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/88e7bc7f-b5f0-472f-838d-9754899a9692-cabundle-cert\") on node \"ip-10-0-138-73.ec2.internal\" DevicePath \"\""
Sep 16 17:43:01.239299 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:43:01.239282    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m4xm9\" (UniqueName: \"kubernetes.io/projected/88e7bc7f-b5f0-472f-838d-9754899a9692-kube-api-access-m4xm9\") on node \"ip-10-0-138-73.ec2.internal\" DevicePath \"\""
Sep 16 17:43:01.239559 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:43:01.239300    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1bdce8d-38c5-4704-b368-f2f2bd274d38-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "b1bdce8d-38c5-4704-b368-f2f2bd274d38" (UID: "b1bdce8d-38c5-4704-b368-f2f2bd274d38"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 17:43:01.239559 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:43:01.239411    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1bdce8d-38c5-4704-b368-f2f2bd274d38-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "b1bdce8d-38c5-4704-b368-f2f2bd274d38" (UID: "b1bdce8d-38c5-4704-b368-f2f2bd274d38"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 17:43:01.239559 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:43:01.239424    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1bdce8d-38c5-4704-b368-f2f2bd274d38-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b1bdce8d-38c5-4704-b368-f2f2bd274d38" (UID: "b1bdce8d-38c5-4704-b368-f2f2bd274d38"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:43:01.241035 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:43:01.241015    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1bdce8d-38c5-4704-b368-f2f2bd274d38-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b1bdce8d-38c5-4704-b368-f2f2bd274d38" (UID: "b1bdce8d-38c5-4704-b368-f2f2bd274d38"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 17:43:01.241103 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:43:01.241065    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1bdce8d-38c5-4704-b368-f2f2bd274d38-kube-api-access-q8djj" (OuterVolumeSpecName: "kube-api-access-q8djj") pod "b1bdce8d-38c5-4704-b368-f2f2bd274d38" (UID: "b1bdce8d-38c5-4704-b368-f2f2bd274d38"). InnerVolumeSpecName "kube-api-access-q8djj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 17:43:01.340607 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:43:01.340568    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b1bdce8d-38c5-4704-b368-f2f2bd274d38-proxy-tls\") on node \"ip-10-0-138-73.ec2.internal\" DevicePath \"\""
Sep 16 17:43:01.340607 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:43:01.340600    2579 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b1bdce8d-38c5-4704-b368-f2f2bd274d38-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-73.ec2.internal\" DevicePath \"\""
Sep 16 17:43:01.340607 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:43:01.340614    2579 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b1bdce8d-38c5-4704-b368-f2f2bd274d38-cabundle-cert\") on node \"ip-10-0-138-73.ec2.internal\" DevicePath \"\""
Sep 16 17:43:01.340882 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:43:01.340626    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q8djj\" (UniqueName: \"kubernetes.io/projected/b1bdce8d-38c5-4704-b368-f2f2bd274d38-kube-api-access-q8djj\") on node \"ip-10-0-138-73.ec2.internal\" DevicePath \"\""
Sep 16 17:43:01.340882 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:43:01.340663    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b1bdce8d-38c5-4704-b368-f2f2bd274d38-kserve-provision-location\") on node \"ip-10-0-138-73.ec2.internal\" DevicePath \"\""
Sep 16 17:43:01.432780 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:43:01.432746    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-5hmxg"]
Sep 16 17:43:01.438112 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:43:01.438083    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-5hmxg"]
Sep 16 17:43:02.098816 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:43:02.098785    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="88e7bc7f-b5f0-472f-838d-9754899a9692" path="/var/lib/kubelet/pods/88e7bc7f-b5f0-472f-838d-9754899a9692/volumes"
Sep 16 17:43:02.119220 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:43:02.119193    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5bb8f495f4-g9h9r"
Sep 16 17:43:02.119348 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:43:02.119221    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5bb8f495f4-g9h9r" event={"ID":"b1bdce8d-38c5-4704-b368-f2f2bd274d38","Type":"ContainerDied","Data":"eced8f5b0376512b46c497bf499d15ae633f105f3d9d984685a45f6409dd79ea"}
Sep 16 17:43:02.119348 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:43:02.119281    2579 scope.go:117] "RemoveContainer" containerID="373aae2492a9a535a43881dd29712fb5c7ee5785ed681e569c929665c4c6d7e4"
Sep 16 17:43:02.127776 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:43:02.127756    2579 scope.go:117] "RemoveContainer" containerID="0217bb35b73abb198db934c75b6e3d99843568049ee2fcb9f8a0c2b70038ce50"
Sep 16 17:43:02.135266 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:43:02.135249    2579 scope.go:117] "RemoveContainer" containerID="b32def8e4c7093d47c34d4a9f8886ec403cf4b42a1cb36e9782f2a991596a625"
Sep 16 17:43:02.137886 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:43:02.137863    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5bb8f495f4-g9h9r"]
Sep 16 17:43:02.141729 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:43:02.141708    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5bb8f495f4-g9h9r"]
Sep 16 17:43:04.098728 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:43:04.098694    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b1bdce8d-38c5-4704-b368-f2f2bd274d38" path="/var/lib/kubelet/pods/b1bdce8d-38c5-4704-b368-f2f2bd274d38/volumes"
Sep 16 17:43:46.774302 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:43:46.774265    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-52ccb-predictor-69557fd477-drkgv"]
Sep 16 17:43:46.774809 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:43:46.774614    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1bdce8d-38c5-4704-b368-f2f2bd274d38" containerName="kube-rbac-proxy"
Sep 16 17:43:46.774809 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:43:46.774625    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1bdce8d-38c5-4704-b368-f2f2bd274d38" containerName="kube-rbac-proxy"
Sep 16 17:43:46.774809 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:43:46.774638    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88e7bc7f-b5f0-472f-838d-9754899a9692" containerName="storage-initializer"
Sep 16 17:43:46.774809 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:43:46.774644    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="88e7bc7f-b5f0-472f-838d-9754899a9692" containerName="storage-initializer"
Sep 16 17:43:46.774809 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:43:46.774654    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88e7bc7f-b5f0-472f-838d-9754899a9692" containerName="kube-rbac-proxy"
Sep 16 17:43:46.774809 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:43:46.774660    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="88e7bc7f-b5f0-472f-838d-9754899a9692" containerName="kube-rbac-proxy"
Sep 16 17:43:46.774809 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:43:46.774669    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="390f6c47-fef4-4d54-b616-e5988767fb0d" containerName="kserve-container"
Sep 16 17:43:46.774809 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:43:46.774675    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="390f6c47-fef4-4d54-b616-e5988767fb0d" containerName="kserve-container"
Sep 16 17:43:46.774809 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:43:46.774681    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88e7bc7f-b5f0-472f-838d-9754899a9692" containerName="kserve-container"
Sep 16 17:43:46.774809 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:43:46.774687    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="88e7bc7f-b5f0-472f-838d-9754899a9692" containerName="kserve-container"
Sep 16 17:43:46.774809 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:43:46.774699    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="390f6c47-fef4-4d54-b616-e5988767fb0d" containerName="kube-rbac-proxy"
Sep 16 17:43:46.774809 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:43:46.774704    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="390f6c47-fef4-4d54-b616-e5988767fb0d" containerName="kube-rbac-proxy"
Sep 16 17:43:46.774809 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:43:46.774714    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1bdce8d-38c5-4704-b368-f2f2bd274d38" containerName="storage-initializer"
Sep 16 17:43:46.774809 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:43:46.774719    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1bdce8d-38c5-4704-b368-f2f2bd274d38" containerName="storage-initializer"
Sep 16 17:43:46.774809 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:43:46.774724    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="390f6c47-fef4-4d54-b616-e5988767fb0d" containerName="storage-initializer"
Sep 16 17:43:46.774809 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:43:46.774730    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="390f6c47-fef4-4d54-b616-e5988767fb0d" containerName="storage-initializer"
Sep 16 17:43:46.774809 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:43:46.774746    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1bdce8d-38c5-4704-b368-f2f2bd274d38" containerName="kserve-container"
Sep 16 17:43:46.774809 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:43:46.774751    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1bdce8d-38c5-4704-b368-f2f2bd274d38" containerName="kserve-container"
Sep 16 17:43:46.774809 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:43:46.774803    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="390f6c47-fef4-4d54-b616-e5988767fb0d" containerName="kube-rbac-proxy"
Sep 16 17:43:46.774809 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:43:46.774809    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="b1bdce8d-38c5-4704-b368-f2f2bd274d38" containerName="kserve-container"
Sep 16 17:43:46.774809 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:43:46.774815    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="390f6c47-fef4-4d54-b616-e5988767fb0d" containerName="kserve-container"
Sep 16 17:43:46.775504 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:43:46.774823    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="88e7bc7f-b5f0-472f-838d-9754899a9692" containerName="kserve-container"
Sep 16 17:43:46.775504 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:43:46.774831    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="88e7bc7f-b5f0-472f-838d-9754899a9692" containerName="kube-rbac-proxy"
Sep 16 17:43:46.775504 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:43:46.774837    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="b1bdce8d-38c5-4704-b368-f2f2bd274d38" containerName="kube-rbac-proxy"
Sep 16 17:43:46.778902 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:43:46.778872    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-52ccb-predictor-69557fd477-drkgv"
Sep 16 17:43:46.782210 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:43:46.782188    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-raw-52ccb-predictor-serving-cert\""
Sep 16 17:43:46.782545 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:43:46.782188    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 16 17:43:46.782669 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:43:46.782197    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-raw-52ccb-kube-rbac-proxy-sar-config\""
Sep 16 17:43:46.783091 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:43:46.783071    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-9bc9t\""
Sep 16 17:43:46.783411 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:43:46.783390    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 16 17:43:46.792090 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:43:46.792067    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-52ccb-predictor-69557fd477-drkgv"]
Sep 16 17:43:46.813493 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:43:46.813445    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqw7l\" (UniqueName: \"kubernetes.io/projected/eb4bd06e-184e-4edb-b672-60157e505d2c-kube-api-access-sqw7l\") pod \"message-dumper-raw-52ccb-predictor-69557fd477-drkgv\" (UID: \"eb4bd06e-184e-4edb-b672-60157e505d2c\") " pod="kserve-ci-e2e-test/message-dumper-raw-52ccb-predictor-69557fd477-drkgv"
Sep 16 17:43:46.813651 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:43:46.813526    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eb4bd06e-184e-4edb-b672-60157e505d2c-proxy-tls\") pod \"message-dumper-raw-52ccb-predictor-69557fd477-drkgv\" (UID: \"eb4bd06e-184e-4edb-b672-60157e505d2c\") " pod="kserve-ci-e2e-test/message-dumper-raw-52ccb-predictor-69557fd477-drkgv"
Sep 16 17:43:46.813651 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:43:46.813580    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eb4bd06e-184e-4edb-b672-60157e505d2c-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-52ccb-predictor-69557fd477-drkgv\" (UID: \"eb4bd06e-184e-4edb-b672-60157e505d2c\") " pod="kserve-ci-e2e-test/message-dumper-raw-52ccb-predictor-69557fd477-drkgv"
Sep 16 17:43:46.914130 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:43:46.914083    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eb4bd06e-184e-4edb-b672-60157e505d2c-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-52ccb-predictor-69557fd477-drkgv\" (UID: \"eb4bd06e-184e-4edb-b672-60157e505d2c\") " pod="kserve-ci-e2e-test/message-dumper-raw-52ccb-predictor-69557fd477-drkgv"
Sep 16 17:43:46.914328 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:43:46.914138    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqw7l\" (UniqueName: \"kubernetes.io/projected/eb4bd06e-184e-4edb-b672-60157e505d2c-kube-api-access-sqw7l\") pod \"message-dumper-raw-52ccb-predictor-69557fd477-drkgv\" (UID: \"eb4bd06e-184e-4edb-b672-60157e505d2c\") " pod="kserve-ci-e2e-test/message-dumper-raw-52ccb-predictor-69557fd477-drkgv"
Sep 16 17:43:46.914328 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:43:46.914198    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eb4bd06e-184e-4edb-b672-60157e505d2c-proxy-tls\") pod \"message-dumper-raw-52ccb-predictor-69557fd477-drkgv\" (UID: \"eb4bd06e-184e-4edb-b672-60157e505d2c\") " pod="kserve-ci-e2e-test/message-dumper-raw-52ccb-predictor-69557fd477-drkgv"
Sep 16 17:43:46.914784 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:43:46.914759    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eb4bd06e-184e-4edb-b672-60157e505d2c-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-52ccb-predictor-69557fd477-drkgv\" (UID: \"eb4bd06e-184e-4edb-b672-60157e505d2c\") " pod="kserve-ci-e2e-test/message-dumper-raw-52ccb-predictor-69557fd477-drkgv"
Sep 16 17:43:46.916613 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:43:46.916588    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eb4bd06e-184e-4edb-b672-60157e505d2c-proxy-tls\") pod \"message-dumper-raw-52ccb-predictor-69557fd477-drkgv\" (UID: \"eb4bd06e-184e-4edb-b672-60157e505d2c\") " pod="kserve-ci-e2e-test/message-dumper-raw-52ccb-predictor-69557fd477-drkgv"
Sep 16 17:43:46.923004 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:43:46.922974    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqw7l\" (UniqueName: \"kubernetes.io/projected/eb4bd06e-184e-4edb-b672-60157e505d2c-kube-api-access-sqw7l\") pod \"message-dumper-raw-52ccb-predictor-69557fd477-drkgv\" (UID: \"eb4bd06e-184e-4edb-b672-60157e505d2c\") " pod="kserve-ci-e2e-test/message-dumper-raw-52ccb-predictor-69557fd477-drkgv"
Sep 16 17:43:47.091248 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:43:47.091206    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-52ccb-predictor-69557fd477-drkgv"
Sep 16 17:43:47.365317 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:43:47.365287    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-52ccb-predictor-69557fd477-drkgv"]
Sep 16 17:43:47.366972 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:43:47.366943    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb4bd06e_184e_4edb_b672_60157e505d2c.slice/crio-a4c3a8fb1c9a5438ad01aa57de92078dc7a257541870cb1a51d94741c39af805 WatchSource:0}: Error finding container a4c3a8fb1c9a5438ad01aa57de92078dc7a257541870cb1a51d94741c39af805: Status 404 returned error can't find the container with id a4c3a8fb1c9a5438ad01aa57de92078dc7a257541870cb1a51d94741c39af805
Sep 16 17:43:47.368976 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:43:47.368961    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 17:43:48.338563 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:43:48.338484    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-52ccb-predictor-69557fd477-drkgv" event={"ID":"eb4bd06e-184e-4edb-b672-60157e505d2c","Type":"ContainerStarted","Data":"a4c3a8fb1c9a5438ad01aa57de92078dc7a257541870cb1a51d94741c39af805"}
Sep 16 17:43:49.344537 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:43:49.344496    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-52ccb-predictor-69557fd477-drkgv" event={"ID":"eb4bd06e-184e-4edb-b672-60157e505d2c","Type":"ContainerStarted","Data":"e8d6bb79581f2fdb959e1800754a889b28cc84c1367ea179438b8e51cbca7f56"}
Sep 16 17:43:49.345045 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:43:49.344564    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-52ccb-predictor-69557fd477-drkgv" event={"ID":"eb4bd06e-184e-4edb-b672-60157e505d2c","Type":"ContainerStarted","Data":"ca83eb53bbbd267fb75d4784196a53cb489765ac0d02d1a78468567bc7d00327"}
Sep 16 17:43:49.345045 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:43:49.344598    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-raw-52ccb-predictor-69557fd477-drkgv"
Sep 16 17:43:49.364030 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:43:49.363977    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/message-dumper-raw-52ccb-predictor-69557fd477-drkgv" podStartSLOduration=2.089057679 podStartE2EDuration="3.363961067s" totalImagesPullingTime="1.274903388s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:43:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:43:49.362688112 +0000 UTC m=+1141.832686247" watchObservedRunningTime="2026-09-16 17:43:49.363961067 +0000 UTC m=+1141.833959202"
Sep 16 17:43:50.349168 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:43:50.349137    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-raw-52ccb-predictor-69557fd477-drkgv"
Sep 16 17:43:50.350998 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:43:50.350975    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-raw-52ccb-predictor-69557fd477-drkgv"
Sep 16 17:43:57.362353 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:43:57.362317    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-raw-52ccb-predictor-69557fd477-drkgv"
Sep 16 17:44:06.839722 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:44:06.839688    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-52ccb-predictor-7bbcdccd74-nt5bw"]
Sep 16 17:44:06.843584 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:44:06.843564    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-52ccb-predictor-7bbcdccd74-nt5bw"
Sep 16 17:44:06.850794 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:44:06.850766    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-raw-52ccb-predictor-serving-cert\""
Sep 16 17:44:06.850932 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:44:06.850900    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 17:44:06.850932 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:44:06.850925    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-raw-52ccb-kube-rbac-proxy-sar-config\""
Sep 16 17:44:06.851381 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:44:06.851366    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 16 17:44:06.867077 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:44:06.867051    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-52ccb-predictor-7bbcdccd74-nt5bw"]
Sep 16 17:44:06.887455 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:44:06.887423    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69bkj\" (UniqueName: \"kubernetes.io/projected/2705ba42-3c59-4fca-aff1-4a4f8affc8e4-kube-api-access-69bkj\") pod \"isvc-logger-raw-52ccb-predictor-7bbcdccd74-nt5bw\" (UID: \"2705ba42-3c59-4fca-aff1-4a4f8affc8e4\") " pod="kserve-ci-e2e-test/isvc-logger-raw-52ccb-predictor-7bbcdccd74-nt5bw"
Sep 16 17:44:06.887584 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:44:06.887467    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2705ba42-3c59-4fca-aff1-4a4f8affc8e4-kserve-provision-location\") pod \"isvc-logger-raw-52ccb-predictor-7bbcdccd74-nt5bw\" (UID: \"2705ba42-3c59-4fca-aff1-4a4f8affc8e4\") " pod="kserve-ci-e2e-test/isvc-logger-raw-52ccb-predictor-7bbcdccd74-nt5bw"
Sep 16 17:44:06.887584 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:44:06.887514    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2705ba42-3c59-4fca-aff1-4a4f8affc8e4-cabundle-cert\") pod \"isvc-logger-raw-52ccb-predictor-7bbcdccd74-nt5bw\" (UID: \"2705ba42-3c59-4fca-aff1-4a4f8affc8e4\") " pod="kserve-ci-e2e-test/isvc-logger-raw-52ccb-predictor-7bbcdccd74-nt5bw"
Sep 16 17:44:06.887584 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:44:06.887546    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2705ba42-3c59-4fca-aff1-4a4f8affc8e4-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-52ccb-predictor-7bbcdccd74-nt5bw\" (UID: \"2705ba42-3c59-4fca-aff1-4a4f8affc8e4\") " pod="kserve-ci-e2e-test/isvc-logger-raw-52ccb-predictor-7bbcdccd74-nt5bw"
Sep 16 17:44:06.887752 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:44:06.887645    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2705ba42-3c59-4fca-aff1-4a4f8affc8e4-proxy-tls\") pod \"isvc-logger-raw-52ccb-predictor-7bbcdccd74-nt5bw\" (UID: \"2705ba42-3c59-4fca-aff1-4a4f8affc8e4\") " pod="kserve-ci-e2e-test/isvc-logger-raw-52ccb-predictor-7bbcdccd74-nt5bw"
Sep 16 17:44:06.988605 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:44:06.988566    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2705ba42-3c59-4fca-aff1-4a4f8affc8e4-kserve-provision-location\") pod \"isvc-logger-raw-52ccb-predictor-7bbcdccd74-nt5bw\" (UID: \"2705ba42-3c59-4fca-aff1-4a4f8affc8e4\") " pod="kserve-ci-e2e-test/isvc-logger-raw-52ccb-predictor-7bbcdccd74-nt5bw"
Sep 16 17:44:06.988605 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:44:06.988607    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2705ba42-3c59-4fca-aff1-4a4f8affc8e4-cabundle-cert\") pod \"isvc-logger-raw-52ccb-predictor-7bbcdccd74-nt5bw\" (UID: \"2705ba42-3c59-4fca-aff1-4a4f8affc8e4\") " pod="kserve-ci-e2e-test/isvc-logger-raw-52ccb-predictor-7bbcdccd74-nt5bw"
Sep 16 17:44:06.988825 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:44:06.988630    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2705ba42-3c59-4fca-aff1-4a4f8affc8e4-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-52ccb-predictor-7bbcdccd74-nt5bw\" (UID: \"2705ba42-3c59-4fca-aff1-4a4f8affc8e4\") " pod="kserve-ci-e2e-test/isvc-logger-raw-52ccb-predictor-7bbcdccd74-nt5bw"
Sep 16 17:44:06.988825 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:44:06.988698    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2705ba42-3c59-4fca-aff1-4a4f8affc8e4-proxy-tls\") pod \"isvc-logger-raw-52ccb-predictor-7bbcdccd74-nt5bw\" (UID: \"2705ba42-3c59-4fca-aff1-4a4f8affc8e4\") " pod="kserve-ci-e2e-test/isvc-logger-raw-52ccb-predictor-7bbcdccd74-nt5bw"
Sep 16 17:44:06.988825 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:44:06.988748    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69bkj\" (UniqueName: \"kubernetes.io/projected/2705ba42-3c59-4fca-aff1-4a4f8affc8e4-kube-api-access-69bkj\") pod \"isvc-logger-raw-52ccb-predictor-7bbcdccd74-nt5bw\" (UID: \"2705ba42-3c59-4fca-aff1-4a4f8affc8e4\") " pod="kserve-ci-e2e-test/isvc-logger-raw-52ccb-predictor-7bbcdccd74-nt5bw"
Sep 16 17:44:06.989079 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:44:06.989058    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2705ba42-3c59-4fca-aff1-4a4f8affc8e4-kserve-provision-location\") pod \"isvc-logger-raw-52ccb-predictor-7bbcdccd74-nt5bw\" (UID: \"2705ba42-3c59-4fca-aff1-4a4f8affc8e4\") " pod="kserve-ci-e2e-test/isvc-logger-raw-52ccb-predictor-7bbcdccd74-nt5bw"
Sep 16 17:44:06.989371 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:44:06.989350    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2705ba42-3c59-4fca-aff1-4a4f8affc8e4-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-52ccb-predictor-7bbcdccd74-nt5bw\" (UID: \"2705ba42-3c59-4fca-aff1-4a4f8affc8e4\") " pod="kserve-ci-e2e-test/isvc-logger-raw-52ccb-predictor-7bbcdccd74-nt5bw"
Sep 16 17:44:06.989428 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:44:06.989356    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2705ba42-3c59-4fca-aff1-4a4f8affc8e4-cabundle-cert\") pod \"isvc-logger-raw-52ccb-predictor-7bbcdccd74-nt5bw\" (UID: \"2705ba42-3c59-4fca-aff1-4a4f8affc8e4\") " pod="kserve-ci-e2e-test/isvc-logger-raw-52ccb-predictor-7bbcdccd74-nt5bw"
Sep 16 17:44:06.991323 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:44:06.991305    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2705ba42-3c59-4fca-aff1-4a4f8affc8e4-proxy-tls\") pod \"isvc-logger-raw-52ccb-predictor-7bbcdccd74-nt5bw\" (UID: \"2705ba42-3c59-4fca-aff1-4a4f8affc8e4\") " pod="kserve-ci-e2e-test/isvc-logger-raw-52ccb-predictor-7bbcdccd74-nt5bw"
Sep 16 17:44:06.997541 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:44:06.997516    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-69bkj\" (UniqueName: \"kubernetes.io/projected/2705ba42-3c59-4fca-aff1-4a4f8affc8e4-kube-api-access-69bkj\") pod \"isvc-logger-raw-52ccb-predictor-7bbcdccd74-nt5bw\" (UID: \"2705ba42-3c59-4fca-aff1-4a4f8affc8e4\") " pod="kserve-ci-e2e-test/isvc-logger-raw-52ccb-predictor-7bbcdccd74-nt5bw"
Sep 16 17:44:07.153729 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:44:07.153639    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-52ccb-predictor-7bbcdccd74-nt5bw"
Sep 16 17:44:07.433387 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:44:07.433365    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-52ccb-predictor-7bbcdccd74-nt5bw"]
Sep 16 17:44:07.435348 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:44:07.435314    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2705ba42_3c59_4fca_aff1_4a4f8affc8e4.slice/crio-3d478468dc3a51643b979adf0d95ccbcc6211616dff3b861a98dd6f7b448b054 WatchSource:0}: Error finding container 3d478468dc3a51643b979adf0d95ccbcc6211616dff3b861a98dd6f7b448b054: Status 404 returned error can't find the container with id 3d478468dc3a51643b979adf0d95ccbcc6211616dff3b861a98dd6f7b448b054
Sep 16 17:44:08.440056 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:44:08.440023    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-52ccb-predictor-7bbcdccd74-nt5bw" event={"ID":"2705ba42-3c59-4fca-aff1-4a4f8affc8e4","Type":"ContainerStarted","Data":"18182c9b514c25c2b376736401a41faaad1ca8b70544e3dce6c20020bf02d8a6"}
Sep 16 17:44:08.440434 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:44:08.440091    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-52ccb-predictor-7bbcdccd74-nt5bw" event={"ID":"2705ba42-3c59-4fca-aff1-4a4f8affc8e4","Type":"ContainerStarted","Data":"3d478468dc3a51643b979adf0d95ccbcc6211616dff3b861a98dd6f7b448b054"}
Sep 16 17:44:11.455920 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:44:11.455883    2579 generic.go:356] "Generic (PLEG): container finished" podID="2705ba42-3c59-4fca-aff1-4a4f8affc8e4" containerID="18182c9b514c25c2b376736401a41faaad1ca8b70544e3dce6c20020bf02d8a6" exitCode=0
Sep 16 17:44:11.456329 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:44:11.455952    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-52ccb-predictor-7bbcdccd74-nt5bw" event={"ID":"2705ba42-3c59-4fca-aff1-4a4f8affc8e4","Type":"ContainerDied","Data":"18182c9b514c25c2b376736401a41faaad1ca8b70544e3dce6c20020bf02d8a6"}
Sep 16 17:44:12.462927 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:44:12.462886    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-52ccb-predictor-7bbcdccd74-nt5bw" event={"ID":"2705ba42-3c59-4fca-aff1-4a4f8affc8e4","Type":"ContainerStarted","Data":"6261435b2a6f8069eaf38ee7018b9be80a8a9b95659f71786daeef03f7f05804"}
Sep 16 17:44:12.463399 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:44:12.462961    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-52ccb-predictor-7bbcdccd74-nt5bw" event={"ID":"2705ba42-3c59-4fca-aff1-4a4f8affc8e4","Type":"ContainerStarted","Data":"37d643fe14d0717da0d865c08dfa71bfc2fff5affc1d89ab56fc6d83bb92d476"}
Sep 16 17:44:12.463399 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:44:12.462988    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-52ccb-predictor-7bbcdccd74-nt5bw" event={"ID":"2705ba42-3c59-4fca-aff1-4a4f8affc8e4","Type":"ContainerStarted","Data":"2631fc546374001e8a661ad740cd36b879710dc654fbe97f50ac94d6fc3bcbb3"}
Sep 16 17:44:12.463399 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:44:12.463242    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-52ccb-predictor-7bbcdccd74-nt5bw"
Sep 16 17:44:12.463399 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:44:12.463314    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-52ccb-predictor-7bbcdccd74-nt5bw"
Sep 16 17:44:12.464827 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:44:12.464797    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-52ccb-predictor-7bbcdccd74-nt5bw" podUID="2705ba42-3c59-4fca-aff1-4a4f8affc8e4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Sep 16 17:44:12.487876 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:44:12.487817    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-logger-raw-52ccb-predictor-7bbcdccd74-nt5bw" podStartSLOduration=2.632138656 podStartE2EDuration="6.487803892s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.855665236s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:44:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:44:12.485499839 +0000 UTC m=+1164.955497969" watchObservedRunningTime="2026-09-16 17:44:12.487803892 +0000 UTC m=+1164.957802028"
Sep 16 17:44:13.469156 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:44:13.469114    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-52ccb-predictor-7bbcdccd74-nt5bw"
Sep 16 17:44:13.469609 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:44:13.469314    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-52ccb-predictor-7bbcdccd74-nt5bw" podUID="2705ba42-3c59-4fca-aff1-4a4f8affc8e4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Sep 16 17:44:13.470556 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:44:13.470528    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-52ccb-predictor-7bbcdccd74-nt5bw" podUID="2705ba42-3c59-4fca-aff1-4a4f8affc8e4" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 17:44:14.474441 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:44:14.474405    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-52ccb-predictor-7bbcdccd74-nt5bw" podUID="2705ba42-3c59-4fca-aff1-4a4f8affc8e4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Sep 16 17:44:14.474922 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:44:14.474789    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-52ccb-predictor-7bbcdccd74-nt5bw" podUID="2705ba42-3c59-4fca-aff1-4a4f8affc8e4" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 17:44:19.480450 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:44:19.480417    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-52ccb-predictor-7bbcdccd74-nt5bw"
Sep 16 17:44:19.480987 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:44:19.480948    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-52ccb-predictor-7bbcdccd74-nt5bw" podUID="2705ba42-3c59-4fca-aff1-4a4f8affc8e4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Sep 16 17:44:19.481441 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:44:19.481416    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-52ccb-predictor-7bbcdccd74-nt5bw" podUID="2705ba42-3c59-4fca-aff1-4a4f8affc8e4" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 17:44:29.481175 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:44:29.481115    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-52ccb-predictor-7bbcdccd74-nt5bw" podUID="2705ba42-3c59-4fca-aff1-4a4f8affc8e4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Sep 16 17:44:29.481750 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:44:29.481698    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-52ccb-predictor-7bbcdccd74-nt5bw" podUID="2705ba42-3c59-4fca-aff1-4a4f8affc8e4" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 17:44:39.481181 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:44:39.481139    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-52ccb-predictor-7bbcdccd74-nt5bw" podUID="2705ba42-3c59-4fca-aff1-4a4f8affc8e4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Sep 16 17:44:39.481619 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:44:39.481593    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-52ccb-predictor-7bbcdccd74-nt5bw" podUID="2705ba42-3c59-4fca-aff1-4a4f8affc8e4" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 17:44:49.481359 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:44:49.481314    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-52ccb-predictor-7bbcdccd74-nt5bw" podUID="2705ba42-3c59-4fca-aff1-4a4f8affc8e4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Sep 16 17:44:49.481801 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:44:49.481693    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-52ccb-predictor-7bbcdccd74-nt5bw" podUID="2705ba42-3c59-4fca-aff1-4a4f8affc8e4" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 17:44:59.481432 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:44:59.481390    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-52ccb-predictor-7bbcdccd74-nt5bw" podUID="2705ba42-3c59-4fca-aff1-4a4f8affc8e4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Sep 16 17:44:59.481931 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:44:59.481882    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-52ccb-predictor-7bbcdccd74-nt5bw" podUID="2705ba42-3c59-4fca-aff1-4a4f8affc8e4" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 17:45:09.481769 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:45:09.481718    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-52ccb-predictor-7bbcdccd74-nt5bw" podUID="2705ba42-3c59-4fca-aff1-4a4f8affc8e4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Sep 16 17:45:09.482311 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:45:09.482180    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-52ccb-predictor-7bbcdccd74-nt5bw" podUID="2705ba42-3c59-4fca-aff1-4a4f8affc8e4" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 17:45:19.481447 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:45:19.481400    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-52ccb-predictor-7bbcdccd74-nt5bw" podUID="2705ba42-3c59-4fca-aff1-4a4f8affc8e4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Sep 16 17:45:19.481970 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:45:19.481876    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-52ccb-predictor-7bbcdccd74-nt5bw" podUID="2705ba42-3c59-4fca-aff1-4a4f8affc8e4" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 17:45:29.481685 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:45:29.481653    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-52ccb-predictor-7bbcdccd74-nt5bw"
Sep 16 17:45:29.484022 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:45:29.481925    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-52ccb-predictor-7bbcdccd74-nt5bw"
Sep 16 17:45:41.827076 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:45:41.826982    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_message-dumper-raw-52ccb-predictor-69557fd477-drkgv_eb4bd06e-184e-4edb-b672-60157e505d2c/kserve-container/0.log"
Sep 16 17:45:42.009609 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:45:42.009574    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-52ccb-predictor-7bbcdccd74-nt5bw"]
Sep 16 17:45:42.009927 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:45:42.009894    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-52ccb-predictor-7bbcdccd74-nt5bw" podUID="2705ba42-3c59-4fca-aff1-4a4f8affc8e4" containerName="kserve-container" containerID="cri-o://2631fc546374001e8a661ad740cd36b879710dc654fbe97f50ac94d6fc3bcbb3" gracePeriod=30
Sep 16 17:45:42.009927 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:45:42.009917    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-52ccb-predictor-7bbcdccd74-nt5bw" podUID="2705ba42-3c59-4fca-aff1-4a4f8affc8e4" containerName="agent" containerID="cri-o://6261435b2a6f8069eaf38ee7018b9be80a8a9b95659f71786daeef03f7f05804" gracePeriod=30
Sep 16 17:45:42.010152 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:45:42.009941    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-52ccb-predictor-7bbcdccd74-nt5bw" podUID="2705ba42-3c59-4fca-aff1-4a4f8affc8e4" containerName="kube-rbac-proxy" containerID="cri-o://37d643fe14d0717da0d865c08dfa71bfc2fff5affc1d89ab56fc6d83bb92d476" gracePeriod=30
Sep 16 17:45:42.056983 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:45:42.056948    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-52ccb-predictor-69557fd477-drkgv"]
Sep 16 17:45:42.057232 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:45:42.057209    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-raw-52ccb-predictor-69557fd477-drkgv" podUID="eb4bd06e-184e-4edb-b672-60157e505d2c" containerName="kserve-container" containerID="cri-o://ca83eb53bbbd267fb75d4784196a53cb489765ac0d02d1a78468567bc7d00327" gracePeriod=30
Sep 16 17:45:42.057354 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:45:42.057322    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-raw-52ccb-predictor-69557fd477-drkgv" podUID="eb4bd06e-184e-4edb-b672-60157e505d2c" containerName="kube-rbac-proxy" containerID="cri-o://e8d6bb79581f2fdb959e1800754a889b28cc84c1367ea179438b8e51cbca7f56" gracePeriod=30
Sep 16 17:45:42.318901 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:45:42.318874    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-52ccb-predictor-69557fd477-drkgv"
Sep 16 17:45:42.407769 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:45:42.407663    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eb4bd06e-184e-4edb-b672-60157e505d2c-kube-rbac-proxy-sar-config\") pod \"eb4bd06e-184e-4edb-b672-60157e505d2c\" (UID: \"eb4bd06e-184e-4edb-b672-60157e505d2c\") "
Sep 16 17:45:42.407769 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:45:42.407740    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sqw7l\" (UniqueName: \"kubernetes.io/projected/eb4bd06e-184e-4edb-b672-60157e505d2c-kube-api-access-sqw7l\") pod \"eb4bd06e-184e-4edb-b672-60157e505d2c\" (UID: \"eb4bd06e-184e-4edb-b672-60157e505d2c\") "
Sep 16 17:45:42.408062 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:45:42.407834    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eb4bd06e-184e-4edb-b672-60157e505d2c-proxy-tls\") pod \"eb4bd06e-184e-4edb-b672-60157e505d2c\" (UID: \"eb4bd06e-184e-4edb-b672-60157e505d2c\") "
Sep 16 17:45:42.408152 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:45:42.408131    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb4bd06e-184e-4edb-b672-60157e505d2c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "eb4bd06e-184e-4edb-b672-60157e505d2c" (UID: "eb4bd06e-184e-4edb-b672-60157e505d2c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 17:45:42.409908 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:45:42.409884    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb4bd06e-184e-4edb-b672-60157e505d2c-kube-api-access-sqw7l" (OuterVolumeSpecName: "kube-api-access-sqw7l") pod "eb4bd06e-184e-4edb-b672-60157e505d2c" (UID: "eb4bd06e-184e-4edb-b672-60157e505d2c"). InnerVolumeSpecName "kube-api-access-sqw7l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 17:45:42.409908 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:45:42.409898    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb4bd06e-184e-4edb-b672-60157e505d2c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "eb4bd06e-184e-4edb-b672-60157e505d2c" (UID: "eb4bd06e-184e-4edb-b672-60157e505d2c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 17:45:42.508533 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:45:42.508490    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eb4bd06e-184e-4edb-b672-60157e505d2c-proxy-tls\") on node \"ip-10-0-138-73.ec2.internal\" DevicePath \"\""
Sep 16 17:45:42.508533 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:45:42.508525    2579 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eb4bd06e-184e-4edb-b672-60157e505d2c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-73.ec2.internal\" DevicePath \"\""
Sep 16 17:45:42.508533 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:45:42.508536    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sqw7l\" (UniqueName: \"kubernetes.io/projected/eb4bd06e-184e-4edb-b672-60157e505d2c-kube-api-access-sqw7l\") on node \"ip-10-0-138-73.ec2.internal\" DevicePath \"\""
Sep 16 17:45:42.913549 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:45:42.913513    2579 generic.go:356] "Generic (PLEG): container finished" podID="eb4bd06e-184e-4edb-b672-60157e505d2c" containerID="e8d6bb79581f2fdb959e1800754a889b28cc84c1367ea179438b8e51cbca7f56" exitCode=2
Sep 16 17:45:42.913549 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:45:42.913543    2579 generic.go:356] "Generic (PLEG): container finished" podID="eb4bd06e-184e-4edb-b672-60157e505d2c" containerID="ca83eb53bbbd267fb75d4784196a53cb489765ac0d02d1a78468567bc7d00327" exitCode=2
Sep 16 17:45:42.914088 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:45:42.913569    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-52ccb-predictor-69557fd477-drkgv"
Sep 16 17:45:42.914088 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:45:42.913560    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-52ccb-predictor-69557fd477-drkgv" event={"ID":"eb4bd06e-184e-4edb-b672-60157e505d2c","Type":"ContainerDied","Data":"e8d6bb79581f2fdb959e1800754a889b28cc84c1367ea179438b8e51cbca7f56"}
Sep 16 17:45:42.914088 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:45:42.913654    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-52ccb-predictor-69557fd477-drkgv" event={"ID":"eb4bd06e-184e-4edb-b672-60157e505d2c","Type":"ContainerDied","Data":"ca83eb53bbbd267fb75d4784196a53cb489765ac0d02d1a78468567bc7d00327"}
Sep 16 17:45:42.914088 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:45:42.913691    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-52ccb-predictor-69557fd477-drkgv" event={"ID":"eb4bd06e-184e-4edb-b672-60157e505d2c","Type":"ContainerDied","Data":"a4c3a8fb1c9a5438ad01aa57de92078dc7a257541870cb1a51d94741c39af805"}
Sep 16 17:45:42.914088 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:45:42.913730    2579 scope.go:117] "RemoveContainer" containerID="e8d6bb79581f2fdb959e1800754a889b28cc84c1367ea179438b8e51cbca7f56"
Sep 16 17:45:42.916840 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:45:42.916817    2579 generic.go:356] "Generic (PLEG): container finished" podID="2705ba42-3c59-4fca-aff1-4a4f8affc8e4" containerID="37d643fe14d0717da0d865c08dfa71bfc2fff5affc1d89ab56fc6d83bb92d476" exitCode=2
Sep 16 17:45:42.916965 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:45:42.916886    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-52ccb-predictor-7bbcdccd74-nt5bw" event={"ID":"2705ba42-3c59-4fca-aff1-4a4f8affc8e4","Type":"ContainerDied","Data":"37d643fe14d0717da0d865c08dfa71bfc2fff5affc1d89ab56fc6d83bb92d476"}
Sep 16 17:45:42.922769 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:45:42.922750    2579 scope.go:117] "RemoveContainer" containerID="ca83eb53bbbd267fb75d4784196a53cb489765ac0d02d1a78468567bc7d00327"
Sep 16 17:45:42.931064 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:45:42.931045    2579 scope.go:117] "RemoveContainer" containerID="e8d6bb79581f2fdb959e1800754a889b28cc84c1367ea179438b8e51cbca7f56"
Sep 16 17:45:42.931361 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:45:42.931343    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8d6bb79581f2fdb959e1800754a889b28cc84c1367ea179438b8e51cbca7f56\": container with ID starting with e8d6bb79581f2fdb959e1800754a889b28cc84c1367ea179438b8e51cbca7f56 not found: ID does not exist" containerID="e8d6bb79581f2fdb959e1800754a889b28cc84c1367ea179438b8e51cbca7f56"
Sep 16 17:45:42.931406 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:45:42.931369    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8d6bb79581f2fdb959e1800754a889b28cc84c1367ea179438b8e51cbca7f56"} err="failed to get container status \"e8d6bb79581f2fdb959e1800754a889b28cc84c1367ea179438b8e51cbca7f56\": rpc error: code = NotFound desc = could not find container \"e8d6bb79581f2fdb959e1800754a889b28cc84c1367ea179438b8e51cbca7f56\": container with ID starting with e8d6bb79581f2fdb959e1800754a889b28cc84c1367ea179438b8e51cbca7f56 not found: ID does not exist"
Sep 16 17:45:42.931406 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:45:42.931388    2579 scope.go:117] "RemoveContainer" containerID="ca83eb53bbbd267fb75d4784196a53cb489765ac0d02d1a78468567bc7d00327"
Sep 16 17:45:42.931650 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:45:42.931635    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca83eb53bbbd267fb75d4784196a53cb489765ac0d02d1a78468567bc7d00327\": container with ID starting with ca83eb53bbbd267fb75d4784196a53cb489765ac0d02d1a78468567bc7d00327 not found: ID does not exist" containerID="ca83eb53bbbd267fb75d4784196a53cb489765ac0d02d1a78468567bc7d00327"
Sep 16 17:45:42.931687 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:45:42.931654    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca83eb53bbbd267fb75d4784196a53cb489765ac0d02d1a78468567bc7d00327"} err="failed to get container status \"ca83eb53bbbd267fb75d4784196a53cb489765ac0d02d1a78468567bc7d00327\": rpc error: code = NotFound desc = could not find container \"ca83eb53bbbd267fb75d4784196a53cb489765ac0d02d1a78468567bc7d00327\": container with ID starting with ca83eb53bbbd267fb75d4784196a53cb489765ac0d02d1a78468567bc7d00327 not found: ID does not exist"
Sep 16 17:45:42.931687 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:45:42.931666    2579 scope.go:117] "RemoveContainer" containerID="e8d6bb79581f2fdb959e1800754a889b28cc84c1367ea179438b8e51cbca7f56"
Sep 16 17:45:42.931929 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:45:42.931900    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8d6bb79581f2fdb959e1800754a889b28cc84c1367ea179438b8e51cbca7f56"} err="failed to get container status \"e8d6bb79581f2fdb959e1800754a889b28cc84c1367ea179438b8e51cbca7f56\": rpc error: code = NotFound desc = could not find container \"e8d6bb79581f2fdb959e1800754a889b28cc84c1367ea179438b8e51cbca7f56\": container with ID starting with e8d6bb79581f2fdb959e1800754a889b28cc84c1367ea179438b8e51cbca7f56 not found: ID does not exist"
Sep 16 17:45:42.931977 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:45:42.931927    2579 scope.go:117] "RemoveContainer" containerID="ca83eb53bbbd267fb75d4784196a53cb489765ac0d02d1a78468567bc7d00327"
Sep 16 17:45:42.932215 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:45:42.932196    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca83eb53bbbd267fb75d4784196a53cb489765ac0d02d1a78468567bc7d00327"} err="failed to get container status \"ca83eb53bbbd267fb75d4784196a53cb489765ac0d02d1a78468567bc7d00327\": rpc error: code = NotFound desc = could not find container \"ca83eb53bbbd267fb75d4784196a53cb489765ac0d02d1a78468567bc7d00327\": container with ID starting with ca83eb53bbbd267fb75d4784196a53cb489765ac0d02d1a78468567bc7d00327 not found: ID does not exist"
Sep 16 17:45:42.938136 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:45:42.938112    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-52ccb-predictor-69557fd477-drkgv"]
Sep 16 17:45:42.941823 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:45:42.941800    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-52ccb-predictor-69557fd477-drkgv"]
Sep 16 17:45:44.100268 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:45:44.100234    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eb4bd06e-184e-4edb-b672-60157e505d2c" path="/var/lib/kubelet/pods/eb4bd06e-184e-4edb-b672-60157e505d2c/volumes"
Sep 16 17:45:44.475250 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:45:44.475162    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-52ccb-predictor-7bbcdccd74-nt5bw" podUID="2705ba42-3c59-4fca-aff1-4a4f8affc8e4" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.42:8643/healthz\": dial tcp 10.134.0.42:8643: connect: connection refused"
Sep 16 17:45:46.940913 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:45:46.940808    2579 generic.go:356] "Generic (PLEG): container finished" podID="2705ba42-3c59-4fca-aff1-4a4f8affc8e4" containerID="2631fc546374001e8a661ad740cd36b879710dc654fbe97f50ac94d6fc3bcbb3" exitCode=0
Sep 16 17:45:46.940913 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:45:46.940882    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-52ccb-predictor-7bbcdccd74-nt5bw" event={"ID":"2705ba42-3c59-4fca-aff1-4a4f8affc8e4","Type":"ContainerDied","Data":"2631fc546374001e8a661ad740cd36b879710dc654fbe97f50ac94d6fc3bcbb3"}
Sep 16 17:45:49.475202 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:45:49.475149    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-52ccb-predictor-7bbcdccd74-nt5bw" podUID="2705ba42-3c59-4fca-aff1-4a4f8affc8e4" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.42:8643/healthz\": dial tcp 10.134.0.42:8643: connect: connection refused"
Sep 16 17:45:49.481614 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:45:49.481577    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-52ccb-predictor-7bbcdccd74-nt5bw" podUID="2705ba42-3c59-4fca-aff1-4a4f8affc8e4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Sep 16 17:45:49.481971 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:45:49.481944    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-52ccb-predictor-7bbcdccd74-nt5bw" podUID="2705ba42-3c59-4fca-aff1-4a4f8affc8e4" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 17:45:54.474599 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:45:54.474551    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-52ccb-predictor-7bbcdccd74-nt5bw" podUID="2705ba42-3c59-4fca-aff1-4a4f8affc8e4" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.42:8643/healthz\": dial tcp 10.134.0.42:8643: connect: connection refused"
Sep 16 17:45:54.475045 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:45:54.474701    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-52ccb-predictor-7bbcdccd74-nt5bw"
Sep 16 17:45:59.474920 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:45:59.474873    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-52ccb-predictor-7bbcdccd74-nt5bw" podUID="2705ba42-3c59-4fca-aff1-4a4f8affc8e4" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.42:8643/healthz\": dial tcp 10.134.0.42:8643: connect: connection refused"
Sep 16 17:45:59.481239 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:45:59.481196    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-52ccb-predictor-7bbcdccd74-nt5bw" podUID="2705ba42-3c59-4fca-aff1-4a4f8affc8e4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Sep 16 17:45:59.481540 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:45:59.481511    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-52ccb-predictor-7bbcdccd74-nt5bw" podUID="2705ba42-3c59-4fca-aff1-4a4f8affc8e4" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 17:46:04.475365 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:46:04.475320    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-52ccb-predictor-7bbcdccd74-nt5bw" podUID="2705ba42-3c59-4fca-aff1-4a4f8affc8e4" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.42:8643/healthz\": dial tcp 10.134.0.42:8643: connect: connection refused"
Sep 16 17:46:09.475188 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:46:09.475138    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-52ccb-predictor-7bbcdccd74-nt5bw" podUID="2705ba42-3c59-4fca-aff1-4a4f8affc8e4" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.42:8643/healthz\": dial tcp 10.134.0.42:8643: connect: connection refused"
Sep 16 17:46:09.481646 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:46:09.481602    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-52ccb-predictor-7bbcdccd74-nt5bw" podUID="2705ba42-3c59-4fca-aff1-4a4f8affc8e4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Sep 16 17:46:09.481808 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:46:09.481752    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-52ccb-predictor-7bbcdccd74-nt5bw"
Sep 16 17:46:09.482043 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:46:09.482020    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-52ccb-predictor-7bbcdccd74-nt5bw" podUID="2705ba42-3c59-4fca-aff1-4a4f8affc8e4" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 17:46:09.482155 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:46:09.482131    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-52ccb-predictor-7bbcdccd74-nt5bw"
Sep 16 17:46:12.067834 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:46:12.067795    2579 generic.go:356] "Generic (PLEG): container finished" podID="2705ba42-3c59-4fca-aff1-4a4f8affc8e4" containerID="6261435b2a6f8069eaf38ee7018b9be80a8a9b95659f71786daeef03f7f05804" exitCode=0
Sep 16 17:46:12.068291 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:46:12.067877    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-52ccb-predictor-7bbcdccd74-nt5bw" event={"ID":"2705ba42-3c59-4fca-aff1-4a4f8affc8e4","Type":"ContainerDied","Data":"6261435b2a6f8069eaf38ee7018b9be80a8a9b95659f71786daeef03f7f05804"}
Sep 16 17:46:12.169355 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:46:12.169331    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-52ccb-predictor-7bbcdccd74-nt5bw"
Sep 16 17:46:12.256420 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:46:12.256379    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2705ba42-3c59-4fca-aff1-4a4f8affc8e4-kube-rbac-proxy-sar-config\") pod \"2705ba42-3c59-4fca-aff1-4a4f8affc8e4\" (UID: \"2705ba42-3c59-4fca-aff1-4a4f8affc8e4\") "
Sep 16 17:46:12.256589 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:46:12.256434    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2705ba42-3c59-4fca-aff1-4a4f8affc8e4-proxy-tls\") pod \"2705ba42-3c59-4fca-aff1-4a4f8affc8e4\" (UID: \"2705ba42-3c59-4fca-aff1-4a4f8affc8e4\") "
Sep 16 17:46:12.256649 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:46:12.256593    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-69bkj\" (UniqueName: \"kubernetes.io/projected/2705ba42-3c59-4fca-aff1-4a4f8affc8e4-kube-api-access-69bkj\") pod \"2705ba42-3c59-4fca-aff1-4a4f8affc8e4\" (UID: \"2705ba42-3c59-4fca-aff1-4a4f8affc8e4\") "
Sep 16 17:46:12.256723 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:46:12.256689    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2705ba42-3c59-4fca-aff1-4a4f8affc8e4-kserve-provision-location\") pod \"2705ba42-3c59-4fca-aff1-4a4f8affc8e4\" (UID: \"2705ba42-3c59-4fca-aff1-4a4f8affc8e4\") "
Sep 16 17:46:12.256785 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:46:12.256730    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2705ba42-3c59-4fca-aff1-4a4f8affc8e4-cabundle-cert\") pod \"2705ba42-3c59-4fca-aff1-4a4f8affc8e4\" (UID: \"2705ba42-3c59-4fca-aff1-4a4f8affc8e4\") "
Sep 16 17:46:12.256840 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:46:12.256824    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2705ba42-3c59-4fca-aff1-4a4f8affc8e4-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "2705ba42-3c59-4fca-aff1-4a4f8affc8e4" (UID: "2705ba42-3c59-4fca-aff1-4a4f8affc8e4"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 17:46:12.257042 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:46:12.257025    2579 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2705ba42-3c59-4fca-aff1-4a4f8affc8e4-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-73.ec2.internal\" DevicePath \"\""
Sep 16 17:46:12.257093 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:46:12.257010    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2705ba42-3c59-4fca-aff1-4a4f8affc8e4-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2705ba42-3c59-4fca-aff1-4a4f8affc8e4" (UID: "2705ba42-3c59-4fca-aff1-4a4f8affc8e4"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:46:12.257189 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:46:12.257160    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2705ba42-3c59-4fca-aff1-4a4f8affc8e4-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "2705ba42-3c59-4fca-aff1-4a4f8affc8e4" (UID: "2705ba42-3c59-4fca-aff1-4a4f8affc8e4"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 17:46:12.258792 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:46:12.258771    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2705ba42-3c59-4fca-aff1-4a4f8affc8e4-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "2705ba42-3c59-4fca-aff1-4a4f8affc8e4" (UID: "2705ba42-3c59-4fca-aff1-4a4f8affc8e4"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 17:46:12.258792 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:46:12.258786    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2705ba42-3c59-4fca-aff1-4a4f8affc8e4-kube-api-access-69bkj" (OuterVolumeSpecName: "kube-api-access-69bkj") pod "2705ba42-3c59-4fca-aff1-4a4f8affc8e4" (UID: "2705ba42-3c59-4fca-aff1-4a4f8affc8e4"). InnerVolumeSpecName "kube-api-access-69bkj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 17:46:12.358154 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:46:12.358123    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2705ba42-3c59-4fca-aff1-4a4f8affc8e4-proxy-tls\") on node \"ip-10-0-138-73.ec2.internal\" DevicePath \"\""
Sep 16 17:46:12.358154 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:46:12.358154    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-69bkj\" (UniqueName: \"kubernetes.io/projected/2705ba42-3c59-4fca-aff1-4a4f8affc8e4-kube-api-access-69bkj\") on node \"ip-10-0-138-73.ec2.internal\" DevicePath \"\""
Sep 16 17:46:12.358154 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:46:12.358163    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2705ba42-3c59-4fca-aff1-4a4f8affc8e4-kserve-provision-location\") on node \"ip-10-0-138-73.ec2.internal\" DevicePath \"\""
Sep 16 17:46:12.358370 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:46:12.358172    2579 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2705ba42-3c59-4fca-aff1-4a4f8affc8e4-cabundle-cert\") on node \"ip-10-0-138-73.ec2.internal\" DevicePath \"\""
Sep 16 17:46:13.075051 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:46:13.075021    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-52ccb-predictor-7bbcdccd74-nt5bw" event={"ID":"2705ba42-3c59-4fca-aff1-4a4f8affc8e4","Type":"ContainerDied","Data":"3d478468dc3a51643b979adf0d95ccbcc6211616dff3b861a98dd6f7b448b054"}
Sep 16 17:46:13.075051 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:46:13.075040    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-52ccb-predictor-7bbcdccd74-nt5bw"
Sep 16 17:46:13.075629 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:46:13.075091    2579 scope.go:117] "RemoveContainer" containerID="6261435b2a6f8069eaf38ee7018b9be80a8a9b95659f71786daeef03f7f05804"
Sep 16 17:46:13.086067 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:46:13.086040    2579 scope.go:117] "RemoveContainer" containerID="37d643fe14d0717da0d865c08dfa71bfc2fff5affc1d89ab56fc6d83bb92d476"
Sep 16 17:46:13.093934 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:46:13.093911    2579 scope.go:117] "RemoveContainer" containerID="2631fc546374001e8a661ad740cd36b879710dc654fbe97f50ac94d6fc3bcbb3"
Sep 16 17:46:13.101189 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:46:13.101145    2579 scope.go:117] "RemoveContainer" containerID="18182c9b514c25c2b376736401a41faaad1ca8b70544e3dce6c20020bf02d8a6"
Sep 16 17:46:13.104677 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:46:13.104657    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-52ccb-predictor-7bbcdccd74-nt5bw"]
Sep 16 17:46:13.116683 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:46:13.115002    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-52ccb-predictor-7bbcdccd74-nt5bw"]
Sep 16 17:46:14.099098 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:46:14.099061    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2705ba42-3c59-4fca-aff1-4a4f8affc8e4" path="/var/lib/kubelet/pods/2705ba42-3c59-4fca-aff1-4a4f8affc8e4/volumes"
Sep 16 17:47:07.654006 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:47:07.653970    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-788ae-predictor-86d57577c4-md8b9"]
Sep 16 17:47:07.654430 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:47:07.654326    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb4bd06e-184e-4edb-b672-60157e505d2c" containerName="kserve-container"
Sep 16 17:47:07.654430 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:47:07.654337    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb4bd06e-184e-4edb-b672-60157e505d2c" containerName="kserve-container"
Sep 16 17:47:07.654430 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:47:07.654347    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2705ba42-3c59-4fca-aff1-4a4f8affc8e4" containerName="kube-rbac-proxy"
Sep 16 17:47:07.654430 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:47:07.654352    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="2705ba42-3c59-4fca-aff1-4a4f8affc8e4" containerName="kube-rbac-proxy"
Sep 16 17:47:07.654430 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:47:07.654363    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2705ba42-3c59-4fca-aff1-4a4f8affc8e4" containerName="storage-initializer"
Sep 16 17:47:07.654430 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:47:07.654369    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="2705ba42-3c59-4fca-aff1-4a4f8affc8e4" containerName="storage-initializer"
Sep 16 17:47:07.654430 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:47:07.654378    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2705ba42-3c59-4fca-aff1-4a4f8affc8e4" containerName="agent"
Sep 16 17:47:07.654430 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:47:07.654384    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="2705ba42-3c59-4fca-aff1-4a4f8affc8e4" containerName="agent"
Sep 16 17:47:07.654430 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:47:07.654395    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb4bd06e-184e-4edb-b672-60157e505d2c" containerName="kube-rbac-proxy"
Sep 16 17:47:07.654430 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:47:07.654401    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb4bd06e-184e-4edb-b672-60157e505d2c" containerName="kube-rbac-proxy"
Sep 16 17:47:07.654430 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:47:07.654408    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2705ba42-3c59-4fca-aff1-4a4f8affc8e4" containerName="kserve-container"
Sep 16 17:47:07.654430 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:47:07.654413    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="2705ba42-3c59-4fca-aff1-4a4f8affc8e4" containerName="kserve-container"
Sep 16 17:47:07.654873 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:47:07.654469    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="eb4bd06e-184e-4edb-b672-60157e505d2c" containerName="kserve-container"
Sep 16 17:47:07.654873 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:47:07.654476    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="eb4bd06e-184e-4edb-b672-60157e505d2c" containerName="kube-rbac-proxy"
Sep 16 17:47:07.654873 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:47:07.654485    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="2705ba42-3c59-4fca-aff1-4a4f8affc8e4" containerName="kube-rbac-proxy"
Sep 16 17:47:07.654873 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:47:07.654491    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="2705ba42-3c59-4fca-aff1-4a4f8affc8e4" containerName="agent"
Sep 16 17:47:07.654873 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:47:07.654498    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="2705ba42-3c59-4fca-aff1-4a4f8affc8e4" containerName="kserve-container"
Sep 16 17:47:07.658998 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:47:07.658976    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-788ae-predictor-86d57577c4-md8b9"
Sep 16 17:47:07.663631 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:47:07.663609    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 16 17:47:07.663795 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:47:07.663772    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 16 17:47:07.664403 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:47:07.664384    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 17:47:07.664684 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:47:07.664661    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 16 17:47:07.664804 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:47:07.664783    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-runtime-788ae-predictor-serving-cert\""
Sep 16 17:47:07.664936 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:47:07.664909    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-runtime-788ae-kube-rbac-proxy-sar-config\""
Sep 16 17:47:07.665072 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:47:07.665054    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-9bc9t\""
Sep 16 17:47:07.674955 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:47:07.674927    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-788ae-predictor-86d57577c4-md8b9"]
Sep 16 17:47:07.731109 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:47:07.731066    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c6be100b-5c75-4ca7-9f98-f9365216b64a-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-788ae-predictor-86d57577c4-md8b9\" (UID: \"c6be100b-5c75-4ca7-9f98-f9365216b64a\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-788ae-predictor-86d57577c4-md8b9"
Sep 16 17:47:07.731109 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:47:07.731107    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c6be100b-5c75-4ca7-9f98-f9365216b64a-kserve-provision-location\") pod \"raw-sklearn-runtime-788ae-predictor-86d57577c4-md8b9\" (UID: \"c6be100b-5c75-4ca7-9f98-f9365216b64a\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-788ae-predictor-86d57577c4-md8b9"
Sep 16 17:47:07.731372 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:47:07.731133    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c6be100b-5c75-4ca7-9f98-f9365216b64a-proxy-tls\") pod \"raw-sklearn-runtime-788ae-predictor-86d57577c4-md8b9\" (UID: \"c6be100b-5c75-4ca7-9f98-f9365216b64a\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-788ae-predictor-86d57577c4-md8b9"
Sep 16 17:47:07.731372 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:47:07.731244    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c6be100b-5c75-4ca7-9f98-f9365216b64a-cabundle-cert\") pod \"raw-sklearn-runtime-788ae-predictor-86d57577c4-md8b9\" (UID: \"c6be100b-5c75-4ca7-9f98-f9365216b64a\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-788ae-predictor-86d57577c4-md8b9"
Sep 16 17:47:07.731372 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:47:07.731303    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzgvt\" (UniqueName: \"kubernetes.io/projected/c6be100b-5c75-4ca7-9f98-f9365216b64a-kube-api-access-jzgvt\") pod \"raw-sklearn-runtime-788ae-predictor-86d57577c4-md8b9\" (UID: \"c6be100b-5c75-4ca7-9f98-f9365216b64a\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-788ae-predictor-86d57577c4-md8b9"
Sep 16 17:47:07.832198 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:47:07.832162    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzgvt\" (UniqueName: \"kubernetes.io/projected/c6be100b-5c75-4ca7-9f98-f9365216b64a-kube-api-access-jzgvt\") pod \"raw-sklearn-runtime-788ae-predictor-86d57577c4-md8b9\" (UID: \"c6be100b-5c75-4ca7-9f98-f9365216b64a\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-788ae-predictor-86d57577c4-md8b9"
Sep 16 17:47:07.832469 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:47:07.832225    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c6be100b-5c75-4ca7-9f98-f9365216b64a-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-788ae-predictor-86d57577c4-md8b9\" (UID: \"c6be100b-5c75-4ca7-9f98-f9365216b64a\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-788ae-predictor-86d57577c4-md8b9"
Sep 16 17:47:07.832469 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:47:07.832246    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c6be100b-5c75-4ca7-9f98-f9365216b64a-kserve-provision-location\") pod \"raw-sklearn-runtime-788ae-predictor-86d57577c4-md8b9\" (UID: \"c6be100b-5c75-4ca7-9f98-f9365216b64a\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-788ae-predictor-86d57577c4-md8b9"
Sep 16 17:47:07.832469 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:47:07.832268    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c6be100b-5c75-4ca7-9f98-f9365216b64a-proxy-tls\") pod \"raw-sklearn-runtime-788ae-predictor-86d57577c4-md8b9\" (UID: \"c6be100b-5c75-4ca7-9f98-f9365216b64a\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-788ae-predictor-86d57577c4-md8b9"
Sep 16 17:47:07.832469 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:47:07.832300    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c6be100b-5c75-4ca7-9f98-f9365216b64a-cabundle-cert\") pod \"raw-sklearn-runtime-788ae-predictor-86d57577c4-md8b9\" (UID: \"c6be100b-5c75-4ca7-9f98-f9365216b64a\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-788ae-predictor-86d57577c4-md8b9"
Sep 16 17:47:07.832732 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:47:07.832713    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c6be100b-5c75-4ca7-9f98-f9365216b64a-kserve-provision-location\") pod \"raw-sklearn-runtime-788ae-predictor-86d57577c4-md8b9\" (UID: \"c6be100b-5c75-4ca7-9f98-f9365216b64a\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-788ae-predictor-86d57577c4-md8b9"
Sep 16 17:47:07.833054 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:47:07.833032    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c6be100b-5c75-4ca7-9f98-f9365216b64a-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-788ae-predictor-86d57577c4-md8b9\" (UID: \"c6be100b-5c75-4ca7-9f98-f9365216b64a\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-788ae-predictor-86d57577c4-md8b9"
Sep 16 17:47:07.833153 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:47:07.833062    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c6be100b-5c75-4ca7-9f98-f9365216b64a-cabundle-cert\") pod \"raw-sklearn-runtime-788ae-predictor-86d57577c4-md8b9\" (UID: \"c6be100b-5c75-4ca7-9f98-f9365216b64a\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-788ae-predictor-86d57577c4-md8b9"
Sep 16 17:47:07.834823 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:47:07.834799    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c6be100b-5c75-4ca7-9f98-f9365216b64a-proxy-tls\") pod \"raw-sklearn-runtime-788ae-predictor-86d57577c4-md8b9\" (UID: \"c6be100b-5c75-4ca7-9f98-f9365216b64a\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-788ae-predictor-86d57577c4-md8b9"
Sep 16 17:47:07.841592 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:47:07.841571    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzgvt\" (UniqueName: \"kubernetes.io/projected/c6be100b-5c75-4ca7-9f98-f9365216b64a-kube-api-access-jzgvt\") pod \"raw-sklearn-runtime-788ae-predictor-86d57577c4-md8b9\" (UID: \"c6be100b-5c75-4ca7-9f98-f9365216b64a\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-788ae-predictor-86d57577c4-md8b9"
Sep 16 17:47:07.969828 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:47:07.969725    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-788ae-predictor-86d57577c4-md8b9"
Sep 16 17:47:08.245422 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:47:08.245384    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-788ae-predictor-86d57577c4-md8b9"]
Sep 16 17:47:08.247574 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:47:08.247549    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6be100b_5c75_4ca7_9f98_f9365216b64a.slice/crio-d04a67403fe08118568827f95d016bcdfe7a0de48670f7c850b9e6e0f8872a4f WatchSource:0}: Error finding container d04a67403fe08118568827f95d016bcdfe7a0de48670f7c850b9e6e0f8872a4f: Status 404 returned error can't find the container with id d04a67403fe08118568827f95d016bcdfe7a0de48670f7c850b9e6e0f8872a4f
Sep 16 17:47:08.331098 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:47:08.331056    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-788ae-predictor-86d57577c4-md8b9" event={"ID":"c6be100b-5c75-4ca7-9f98-f9365216b64a","Type":"ContainerStarted","Data":"74c734edbd6eca180a5df68c9b946fbb8b53cb4395f1f9dea7c616daa322dfd0"}
Sep 16 17:47:08.331279 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:47:08.331137    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-788ae-predictor-86d57577c4-md8b9" event={"ID":"c6be100b-5c75-4ca7-9f98-f9365216b64a","Type":"ContainerStarted","Data":"d04a67403fe08118568827f95d016bcdfe7a0de48670f7c850b9e6e0f8872a4f"}
Sep 16 17:47:12.350869 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:47:12.350815    2579 generic.go:356] "Generic (PLEG): container finished" podID="c6be100b-5c75-4ca7-9f98-f9365216b64a" containerID="74c734edbd6eca180a5df68c9b946fbb8b53cb4395f1f9dea7c616daa322dfd0" exitCode=0
Sep 16 17:47:12.351236 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:47:12.350884    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-788ae-predictor-86d57577c4-md8b9" event={"ID":"c6be100b-5c75-4ca7-9f98-f9365216b64a","Type":"ContainerDied","Data":"74c734edbd6eca180a5df68c9b946fbb8b53cb4395f1f9dea7c616daa322dfd0"}
Sep 16 17:47:13.357652 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:47:13.357616    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-788ae-predictor-86d57577c4-md8b9" event={"ID":"c6be100b-5c75-4ca7-9f98-f9365216b64a","Type":"ContainerStarted","Data":"a77d24a3118185ea322cceebf8a3fe38362a27ede0939adf1752f4d4cee24a6b"}
Sep 16 17:47:13.358104 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:47:13.357682    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-788ae-predictor-86d57577c4-md8b9" event={"ID":"c6be100b-5c75-4ca7-9f98-f9365216b64a","Type":"ContainerStarted","Data":"3aa520893be04587590f42200eaf2972ea053f619010f3dfe490d9bd200ecd1a"}
Sep 16 17:47:13.358104 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:47:13.357899    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-788ae-predictor-86d57577c4-md8b9"
Sep 16 17:47:13.385217 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:47:13.385161    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/raw-sklearn-runtime-788ae-predictor-86d57577c4-md8b9" podStartSLOduration=2.462653111 podStartE2EDuration="6.385145506s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.922492395s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:47:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:47:13.38286634 +0000 UTC m=+1345.852864472" watchObservedRunningTime="2026-09-16 17:47:13.385145506 +0000 UTC m=+1345.855143638"
Sep 16 17:47:14.362374 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:47:14.362335    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-788ae-predictor-86d57577c4-md8b9"
Sep 16 17:47:14.363898 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:47:14.363866    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-788ae-predictor-86d57577c4-md8b9" podUID="c6be100b-5c75-4ca7-9f98-f9365216b64a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Sep 16 17:47:15.367995 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:47:15.367948    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-788ae-predictor-86d57577c4-md8b9" podUID="c6be100b-5c75-4ca7-9f98-f9365216b64a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Sep 16 17:47:20.372825 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:47:20.372790    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-788ae-predictor-86d57577c4-md8b9"
Sep 16 17:47:20.373334 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:47:20.373305    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-788ae-predictor-86d57577c4-md8b9" podUID="c6be100b-5c75-4ca7-9f98-f9365216b64a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Sep 16 17:47:30.373469 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:47:30.373426    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-788ae-predictor-86d57577c4-md8b9" podUID="c6be100b-5c75-4ca7-9f98-f9365216b64a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Sep 16 17:47:40.374327 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:47:40.374278    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-788ae-predictor-86d57577c4-md8b9" podUID="c6be100b-5c75-4ca7-9f98-f9365216b64a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Sep 16 17:47:50.374197 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:47:50.374152    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-788ae-predictor-86d57577c4-md8b9" podUID="c6be100b-5c75-4ca7-9f98-f9365216b64a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Sep 16 17:48:00.373964 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:48:00.373928    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-788ae-predictor-86d57577c4-md8b9" podUID="c6be100b-5c75-4ca7-9f98-f9365216b64a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Sep 16 17:48:10.373775 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:48:10.373721    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-788ae-predictor-86d57577c4-md8b9" podUID="c6be100b-5c75-4ca7-9f98-f9365216b64a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Sep 16 17:48:20.373337 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:48:20.373297    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-788ae-predictor-86d57577c4-md8b9" podUID="c6be100b-5c75-4ca7-9f98-f9365216b64a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Sep 16 17:48:30.374005 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:48:30.373963    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-788ae-predictor-86d57577c4-md8b9"
Sep 16 17:48:37.930526 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:48:37.930493    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-788ae-predictor-86d57577c4-md8b9"]
Sep 16 17:48:37.931052 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:48:37.930731    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-runtime-788ae-predictor-86d57577c4-md8b9" podUID="c6be100b-5c75-4ca7-9f98-f9365216b64a" containerName="kserve-container" containerID="cri-o://3aa520893be04587590f42200eaf2972ea053f619010f3dfe490d9bd200ecd1a" gracePeriod=30
Sep 16 17:48:37.931052 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:48:37.930770    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-runtime-788ae-predictor-86d57577c4-md8b9" podUID="c6be100b-5c75-4ca7-9f98-f9365216b64a" containerName="kube-rbac-proxy" containerID="cri-o://a77d24a3118185ea322cceebf8a3fe38362a27ede0939adf1752f4d4cee24a6b" gracePeriod=30
Sep 16 17:48:38.773185 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:48:38.773150    2579 generic.go:356] "Generic (PLEG): container finished" podID="c6be100b-5c75-4ca7-9f98-f9365216b64a" containerID="a77d24a3118185ea322cceebf8a3fe38362a27ede0939adf1752f4d4cee24a6b" exitCode=2
Sep 16 17:48:38.773371 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:48:38.773220    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-788ae-predictor-86d57577c4-md8b9" event={"ID":"c6be100b-5c75-4ca7-9f98-f9365216b64a","Type":"ContainerDied","Data":"a77d24a3118185ea322cceebf8a3fe38362a27ede0939adf1752f4d4cee24a6b"}
Sep 16 17:48:40.368379 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:48:40.368288    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-788ae-predictor-86d57577c4-md8b9" podUID="c6be100b-5c75-4ca7-9f98-f9365216b64a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.43:8643/healthz\": dial tcp 10.134.0.43:8643: connect: connection refused"
Sep 16 17:48:40.373670 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:48:40.373640    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-788ae-predictor-86d57577c4-md8b9" podUID="c6be100b-5c75-4ca7-9f98-f9365216b64a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Sep 16 17:48:42.794547 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:48:42.794508    2579 generic.go:356] "Generic (PLEG): container finished" podID="c6be100b-5c75-4ca7-9f98-f9365216b64a" containerID="3aa520893be04587590f42200eaf2972ea053f619010f3dfe490d9bd200ecd1a" exitCode=0
Sep 16 17:48:42.794932 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:48:42.794541    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-788ae-predictor-86d57577c4-md8b9" event={"ID":"c6be100b-5c75-4ca7-9f98-f9365216b64a","Type":"ContainerDied","Data":"3aa520893be04587590f42200eaf2972ea053f619010f3dfe490d9bd200ecd1a"}
Sep 16 17:48:42.880619 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:48:42.880588    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-788ae-predictor-86d57577c4-md8b9"
Sep 16 17:48:42.957989 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:48:42.957900    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jzgvt\" (UniqueName: \"kubernetes.io/projected/c6be100b-5c75-4ca7-9f98-f9365216b64a-kube-api-access-jzgvt\") pod \"c6be100b-5c75-4ca7-9f98-f9365216b64a\" (UID: \"c6be100b-5c75-4ca7-9f98-f9365216b64a\") "
Sep 16 17:48:42.957989 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:48:42.957943    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c6be100b-5c75-4ca7-9f98-f9365216b64a-kube-rbac-proxy-sar-config\") pod \"c6be100b-5c75-4ca7-9f98-f9365216b64a\" (UID: \"c6be100b-5c75-4ca7-9f98-f9365216b64a\") "
Sep 16 17:48:42.958234 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:48:42.958006    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c6be100b-5c75-4ca7-9f98-f9365216b64a-cabundle-cert\") pod \"c6be100b-5c75-4ca7-9f98-f9365216b64a\" (UID: \"c6be100b-5c75-4ca7-9f98-f9365216b64a\") "
Sep 16 17:48:42.958234 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:48:42.958035    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c6be100b-5c75-4ca7-9f98-f9365216b64a-kserve-provision-location\") pod \"c6be100b-5c75-4ca7-9f98-f9365216b64a\" (UID: \"c6be100b-5c75-4ca7-9f98-f9365216b64a\") "
Sep 16 17:48:42.958234 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:48:42.958094    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c6be100b-5c75-4ca7-9f98-f9365216b64a-proxy-tls\") pod \"c6be100b-5c75-4ca7-9f98-f9365216b64a\" (UID: \"c6be100b-5c75-4ca7-9f98-f9365216b64a\") "
Sep 16 17:48:42.958431 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:48:42.958396    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6be100b-5c75-4ca7-9f98-f9365216b64a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "c6be100b-5c75-4ca7-9f98-f9365216b64a" (UID: "c6be100b-5c75-4ca7-9f98-f9365216b64a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 17:48:42.958595 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:48:42.958566    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6be100b-5c75-4ca7-9f98-f9365216b64a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c6be100b-5c75-4ca7-9f98-f9365216b64a" (UID: "c6be100b-5c75-4ca7-9f98-f9365216b64a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:48:42.958776 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:48:42.958757    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6be100b-5c75-4ca7-9f98-f9365216b64a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "c6be100b-5c75-4ca7-9f98-f9365216b64a" (UID: "c6be100b-5c75-4ca7-9f98-f9365216b64a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 17:48:42.960566 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:48:42.960538    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6be100b-5c75-4ca7-9f98-f9365216b64a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c6be100b-5c75-4ca7-9f98-f9365216b64a" (UID: "c6be100b-5c75-4ca7-9f98-f9365216b64a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 17:48:42.960688 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:48:42.960638    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6be100b-5c75-4ca7-9f98-f9365216b64a-kube-api-access-jzgvt" (OuterVolumeSpecName: "kube-api-access-jzgvt") pod "c6be100b-5c75-4ca7-9f98-f9365216b64a" (UID: "c6be100b-5c75-4ca7-9f98-f9365216b64a"). InnerVolumeSpecName "kube-api-access-jzgvt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 17:48:43.059693 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:48:43.059660    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c6be100b-5c75-4ca7-9f98-f9365216b64a-proxy-tls\") on node \"ip-10-0-138-73.ec2.internal\" DevicePath \"\""
Sep 16 17:48:43.059693 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:48:43.059691    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jzgvt\" (UniqueName: \"kubernetes.io/projected/c6be100b-5c75-4ca7-9f98-f9365216b64a-kube-api-access-jzgvt\") on node \"ip-10-0-138-73.ec2.internal\" DevicePath \"\""
Sep 16 17:48:43.059693 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:48:43.059701    2579 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c6be100b-5c75-4ca7-9f98-f9365216b64a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-73.ec2.internal\" DevicePath \"\""
Sep 16 17:48:43.059963 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:48:43.059710    2579 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c6be100b-5c75-4ca7-9f98-f9365216b64a-cabundle-cert\") on node \"ip-10-0-138-73.ec2.internal\" DevicePath \"\""
Sep 16 17:48:43.059963 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:48:43.059720    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c6be100b-5c75-4ca7-9f98-f9365216b64a-kserve-provision-location\") on node \"ip-10-0-138-73.ec2.internal\" DevicePath \"\""
Sep 16 17:48:43.103400 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:48:43.103369    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5cd44d6886-4v9nb"]
Sep 16 17:48:43.103762 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:48:43.103749    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6be100b-5c75-4ca7-9f98-f9365216b64a" containerName="storage-initializer"
Sep 16 17:48:43.103811 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:48:43.103764    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6be100b-5c75-4ca7-9f98-f9365216b64a" containerName="storage-initializer"
Sep 16 17:48:43.103811 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:48:43.103774    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6be100b-5c75-4ca7-9f98-f9365216b64a" containerName="kube-rbac-proxy"
Sep 16 17:48:43.103811 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:48:43.103779    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6be100b-5c75-4ca7-9f98-f9365216b64a" containerName="kube-rbac-proxy"
Sep 16 17:48:43.103811 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:48:43.103797    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6be100b-5c75-4ca7-9f98-f9365216b64a" containerName="kserve-container"
Sep 16 17:48:43.103811 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:48:43.103806    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6be100b-5c75-4ca7-9f98-f9365216b64a" containerName="kserve-container"
Sep 16 17:48:43.103982 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:48:43.103882    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="c6be100b-5c75-4ca7-9f98-f9365216b64a" containerName="kserve-container"
Sep 16 17:48:43.103982 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:48:43.103892    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="c6be100b-5c75-4ca7-9f98-f9365216b64a" containerName="kube-rbac-proxy"
Sep 16 17:48:43.107090 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:48:43.107072    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5cd44d6886-4v9nb"
Sep 16 17:48:43.110461 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:48:43.110428    2579 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"isvc-canary-create-kube-rbac-proxy-sar-config\" is forbidden: User \"system:node:ip-10-0-138-73.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"kserve-ci-e2e-test\": no relationship found between node 'ip-10-0-138-73.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-kube-rbac-proxy-sar-config\"" type="*v1.ConfigMap"
Sep 16 17:48:43.110579 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:48:43.110529    2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"isvc-canary-create-v2-predictor-5cd44d6886-4v9nb\" is forbidden: User \"system:node:ip-10-0-138-73.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kserve-ci-e2e-test\": no relationship found between node 'ip-10-0-138-73.ec2.internal' and this object" podUID="4f6a368c-bc67-4bda-9fd6-554d4e20d681" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5cd44d6886-4v9nb"
Sep 16 17:48:43.110629 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:48:43.110581    2579 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"isvc-canary-create-v2-predictor-serving-cert\" is forbidden: User \"system:node:ip-10-0-138-73.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"kserve-ci-e2e-test\": no relationship found between node 'ip-10-0-138-73.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-v2-predictor-serving-cert\"" type="*v1.Secret"
Sep 16 17:48:43.125216 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:48:43.125186    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-npzlr"]
Sep 16 17:48:43.128790 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:48:43.128769    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5cd44d6886-4v9nb"]
Sep 16 17:48:43.128926 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:48:43.128882    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-npzlr"
Sep 16 17:48:43.136997 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:48:43.136972    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-predictor-serving-cert\""
Sep 16 17:48:43.147484 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:48:43.147457    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-npzlr"]
Sep 16 17:48:43.160814 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:48:43.160785    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4f6a368c-bc67-4bda-9fd6-554d4e20d681-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-5cd44d6886-4v9nb\" (UID: \"4f6a368c-bc67-4bda-9fd6-554d4e20d681\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5cd44d6886-4v9nb"
Sep 16 17:48:43.160985 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:48:43.160823    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4f6a368c-bc67-4bda-9fd6-554d4e20d681-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-5cd44d6886-4v9nb\" (UID: \"4f6a368c-bc67-4bda-9fd6-554d4e20d681\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5cd44d6886-4v9nb"
Sep 16 17:48:43.160985 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:48:43.160878    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/edfe795b-2f83-445b-95de-bfb096759517-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-854596548c-npzlr\" (UID: \"edfe795b-2f83-445b-95de-bfb096759517\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-npzlr"
Sep 16 17:48:43.160985 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:48:43.160934    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/edfe795b-2f83-445b-95de-bfb096759517-proxy-tls\") pod \"isvc-canary-create-predictor-854596548c-npzlr\" (UID: \"edfe795b-2f83-445b-95de-bfb096759517\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-npzlr"
Sep 16 17:48:43.161164 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:48:43.161047    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vgzg\" (UniqueName: \"kubernetes.io/projected/4f6a368c-bc67-4bda-9fd6-554d4e20d681-kube-api-access-4vgzg\") pod \"isvc-canary-create-v2-predictor-5cd44d6886-4v9nb\" (UID: \"4f6a368c-bc67-4bda-9fd6-554d4e20d681\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5cd44d6886-4v9nb"
Sep 16 17:48:43.161164 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:48:43.161072    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmch2\" (UniqueName: \"kubernetes.io/projected/edfe795b-2f83-445b-95de-bfb096759517-kube-api-access-zmch2\") pod \"isvc-canary-create-predictor-854596548c-npzlr\" (UID: \"edfe795b-2f83-445b-95de-bfb096759517\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-npzlr"
Sep 16 17:48:43.161164 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:48:43.161090    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/edfe795b-2f83-445b-95de-bfb096759517-cabundle-cert\") pod \"isvc-canary-create-predictor-854596548c-npzlr\" (UID: \"edfe795b-2f83-445b-95de-bfb096759517\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-npzlr"
Sep 16 17:48:43.161164 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:48:43.161106    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4f6a368c-bc67-4bda-9fd6-554d4e20d681-proxy-tls\") pod \"isvc-canary-create-v2-predictor-5cd44d6886-4v9nb\" (UID: \"4f6a368c-bc67-4bda-9fd6-554d4e20d681\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5cd44d6886-4v9nb"
Sep 16 17:48:43.161164 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:48:43.161129    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/edfe795b-2f83-445b-95de-bfb096759517-kserve-provision-location\") pod \"isvc-canary-create-predictor-854596548c-npzlr\" (UID: \"edfe795b-2f83-445b-95de-bfb096759517\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-npzlr"
Sep 16 17:48:43.161443 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:48:43.161190    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4f6a368c-bc67-4bda-9fd6-554d4e20d681-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-5cd44d6886-4v9nb\" (UID: \"4f6a368c-bc67-4bda-9fd6-554d4e20d681\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5cd44d6886-4v9nb"
Sep 16 17:48:43.262418 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:48:43.262332    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4f6a368c-bc67-4bda-9fd6-554d4e20d681-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-5cd44d6886-4v9nb\" (UID: \"4f6a368c-bc67-4bda-9fd6-554d4e20d681\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5cd44d6886-4v9nb"
Sep 16 17:48:43.262418 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:48:43.262374    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4f6a368c-bc67-4bda-9fd6-554d4e20d681-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-5cd44d6886-4v9nb\" (UID: \"4f6a368c-bc67-4bda-9fd6-554d4e20d681\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5cd44d6886-4v9nb"
Sep 16 17:48:43.262609 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:48:43.262469    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4f6a368c-bc67-4bda-9fd6-554d4e20d681-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-5cd44d6886-4v9nb\" (UID: \"4f6a368c-bc67-4bda-9fd6-554d4e20d681\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5cd44d6886-4v9nb"
Sep 16 17:48:43.262609 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:48:43.262513    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/edfe795b-2f83-445b-95de-bfb096759517-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-854596548c-npzlr\" (UID: \"edfe795b-2f83-445b-95de-bfb096759517\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-npzlr"
Sep 16 17:48:43.262609 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:48:43.262591    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/edfe795b-2f83-445b-95de-bfb096759517-proxy-tls\") pod \"isvc-canary-create-predictor-854596548c-npzlr\" (UID: \"edfe795b-2f83-445b-95de-bfb096759517\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-npzlr"
Sep 16 17:48:43.262764 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:48:43.262622    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vgzg\" (UniqueName: \"kubernetes.io/projected/4f6a368c-bc67-4bda-9fd6-554d4e20d681-kube-api-access-4vgzg\") pod \"isvc-canary-create-v2-predictor-5cd44d6886-4v9nb\" (UID: \"4f6a368c-bc67-4bda-9fd6-554d4e20d681\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5cd44d6886-4v9nb"
Sep 16 17:48:43.262764 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:48:43.262658    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmch2\" (UniqueName: \"kubernetes.io/projected/edfe795b-2f83-445b-95de-bfb096759517-kube-api-access-zmch2\") pod \"isvc-canary-create-predictor-854596548c-npzlr\" (UID: \"edfe795b-2f83-445b-95de-bfb096759517\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-npzlr"
Sep 16 17:48:43.262764 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:48:43.262679    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/edfe795b-2f83-445b-95de-bfb096759517-cabundle-cert\") pod \"isvc-canary-create-predictor-854596548c-npzlr\" (UID: \"edfe795b-2f83-445b-95de-bfb096759517\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-npzlr"
Sep 16 17:48:43.262764 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:48:43.262696    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4f6a368c-bc67-4bda-9fd6-554d4e20d681-proxy-tls\") pod \"isvc-canary-create-v2-predictor-5cd44d6886-4v9nb\" (UID: \"4f6a368c-bc67-4bda-9fd6-554d4e20d681\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5cd44d6886-4v9nb"
Sep 16 17:48:43.262764 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:48:43.262723    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/edfe795b-2f83-445b-95de-bfb096759517-kserve-provision-location\") pod \"isvc-canary-create-predictor-854596548c-npzlr\" (UID: \"edfe795b-2f83-445b-95de-bfb096759517\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-npzlr"
Sep 16 17:48:43.262764 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:48:43.262752    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4f6a368c-bc67-4bda-9fd6-554d4e20d681-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-5cd44d6886-4v9nb\" (UID: \"4f6a368c-bc67-4bda-9fd6-554d4e20d681\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5cd44d6886-4v9nb"
Sep 16 17:48:43.263195 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:48:43.263175    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/edfe795b-2f83-445b-95de-bfb096759517-kserve-provision-location\") pod \"isvc-canary-create-predictor-854596548c-npzlr\" (UID: \"edfe795b-2f83-445b-95de-bfb096759517\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-npzlr"
Sep 16 17:48:43.263248 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:48:43.263227    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4f6a368c-bc67-4bda-9fd6-554d4e20d681-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-5cd44d6886-4v9nb\" (UID: \"4f6a368c-bc67-4bda-9fd6-554d4e20d681\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5cd44d6886-4v9nb"
Sep 16 17:48:43.263341 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:48:43.263324    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/edfe795b-2f83-445b-95de-bfb096759517-cabundle-cert\") pod \"isvc-canary-create-predictor-854596548c-npzlr\" (UID: \"edfe795b-2f83-445b-95de-bfb096759517\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-npzlr"
Sep 16 17:48:43.265114 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:48:43.265092    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/edfe795b-2f83-445b-95de-bfb096759517-proxy-tls\") pod \"isvc-canary-create-predictor-854596548c-npzlr\" (UID: \"edfe795b-2f83-445b-95de-bfb096759517\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-npzlr"
Sep 16 17:48:43.276268 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:48:43.276247    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmch2\" (UniqueName: \"kubernetes.io/projected/edfe795b-2f83-445b-95de-bfb096759517-kube-api-access-zmch2\") pod \"isvc-canary-create-predictor-854596548c-npzlr\" (UID: \"edfe795b-2f83-445b-95de-bfb096759517\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-npzlr"
Sep 16 17:48:43.282062 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:48:43.282044    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vgzg\" (UniqueName: \"kubernetes.io/projected/4f6a368c-bc67-4bda-9fd6-554d4e20d681-kube-api-access-4vgzg\") pod \"isvc-canary-create-v2-predictor-5cd44d6886-4v9nb\" (UID: \"4f6a368c-bc67-4bda-9fd6-554d4e20d681\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5cd44d6886-4v9nb"
Sep 16 17:48:43.801465 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:48:43.801435    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-788ae-predictor-86d57577c4-md8b9"
Sep 16 17:48:43.801465 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:48:43.801446    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-788ae-predictor-86d57577c4-md8b9" event={"ID":"c6be100b-5c75-4ca7-9f98-f9365216b64a","Type":"ContainerDied","Data":"d04a67403fe08118568827f95d016bcdfe7a0de48670f7c850b9e6e0f8872a4f"}
Sep 16 17:48:43.801971 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:48:43.801514    2579 scope.go:117] "RemoveContainer" containerID="a77d24a3118185ea322cceebf8a3fe38362a27ede0939adf1752f4d4cee24a6b"
Sep 16 17:48:43.810972 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:48:43.810952    2579 scope.go:117] "RemoveContainer" containerID="3aa520893be04587590f42200eaf2972ea053f619010f3dfe490d9bd200ecd1a"
Sep 16 17:48:43.819300 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:48:43.819278    2579 scope.go:117] "RemoveContainer" containerID="74c734edbd6eca180a5df68c9b946fbb8b53cb4395f1f9dea7c616daa322dfd0"
Sep 16 17:48:43.839439 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:48:43.839394    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-788ae-predictor-86d57577c4-md8b9"]
Sep 16 17:48:43.844136 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:48:43.844110    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-788ae-predictor-86d57577c4-md8b9"]
Sep 16 17:48:44.099183 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:48:44.099107    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c6be100b-5c75-4ca7-9f98-f9365216b64a" path="/var/lib/kubelet/pods/c6be100b-5c75-4ca7-9f98-f9365216b64a/volumes"
Sep 16 17:48:44.262642 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:48:44.262611    2579 configmap.go:193] Couldn't get configMap kserve-ci-e2e-test/isvc-canary-create-kube-rbac-proxy-sar-config: failed to sync configmap cache: timed out waiting for the condition
Sep 16 17:48:44.262780 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:48:44.262690    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4f6a368c-bc67-4bda-9fd6-554d4e20d681-kube-rbac-proxy-sar-config podName:4f6a368c-bc67-4bda-9fd6-554d4e20d681 nodeName:}" failed. No retries permitted until 2026-09-16 17:48:44.762671896 +0000 UTC m=+1437.232670008 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-sar-config" (UniqueName: "kubernetes.io/configmap/4f6a368c-bc67-4bda-9fd6-554d4e20d681-kube-rbac-proxy-sar-config") pod "isvc-canary-create-v2-predictor-5cd44d6886-4v9nb" (UID: "4f6a368c-bc67-4bda-9fd6-554d4e20d681") : failed to sync configmap cache: timed out waiting for the condition
Sep 16 17:48:44.262864 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:48:44.262778    2579 configmap.go:193] Couldn't get configMap kserve-ci-e2e-test/isvc-canary-create-kube-rbac-proxy-sar-config: failed to sync configmap cache: timed out waiting for the condition
Sep 16 17:48:44.262864 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:48:44.262833    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/edfe795b-2f83-445b-95de-bfb096759517-kube-rbac-proxy-sar-config podName:edfe795b-2f83-445b-95de-bfb096759517 nodeName:}" failed. No retries permitted until 2026-09-16 17:48:44.762819405 +0000 UTC m=+1437.232817518 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-sar-config" (UniqueName: "kubernetes.io/configmap/edfe795b-2f83-445b-95de-bfb096759517-kube-rbac-proxy-sar-config") pod "isvc-canary-create-predictor-854596548c-npzlr" (UID: "edfe795b-2f83-445b-95de-bfb096759517") : failed to sync configmap cache: timed out waiting for the condition
Sep 16 17:48:44.262947 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:48:44.262919    2579 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-create-v2-predictor-serving-cert: failed to sync secret cache: timed out waiting for the condition
Sep 16 17:48:44.262987 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:48:44.262956    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4f6a368c-bc67-4bda-9fd6-554d4e20d681-proxy-tls podName:4f6a368c-bc67-4bda-9fd6-554d4e20d681 nodeName:}" failed. No retries permitted until 2026-09-16 17:48:44.762946747 +0000 UTC m=+1437.232944860 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/4f6a368c-bc67-4bda-9fd6-554d4e20d681-proxy-tls") pod "isvc-canary-create-v2-predictor-5cd44d6886-4v9nb" (UID: "4f6a368c-bc67-4bda-9fd6-554d4e20d681") : failed to sync secret cache: timed out waiting for the condition
Sep 16 17:48:44.348538 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:48:44.348508    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-v2-predictor-serving-cert\""
Sep 16 17:48:44.411530 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:48:44.411450    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-kube-rbac-proxy-sar-config\""
Sep 16 17:48:44.776762 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:48:44.776662    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4f6a368c-bc67-4bda-9fd6-554d4e20d681-proxy-tls\") pod \"isvc-canary-create-v2-predictor-5cd44d6886-4v9nb\" (UID: \"4f6a368c-bc67-4bda-9fd6-554d4e20d681\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5cd44d6886-4v9nb"
Sep 16 17:48:44.776762 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:48:44.776719    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4f6a368c-bc67-4bda-9fd6-554d4e20d681-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-5cd44d6886-4v9nb\" (UID: \"4f6a368c-bc67-4bda-9fd6-554d4e20d681\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5cd44d6886-4v9nb"
Sep 16 17:48:44.776762 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:48:44.776757    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/edfe795b-2f83-445b-95de-bfb096759517-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-854596548c-npzlr\" (UID: \"edfe795b-2f83-445b-95de-bfb096759517\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-npzlr"
Sep 16 17:48:44.777359 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:48:44.777328    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/edfe795b-2f83-445b-95de-bfb096759517-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-854596548c-npzlr\" (UID: \"edfe795b-2f83-445b-95de-bfb096759517\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-npzlr"
Sep 16 17:48:44.777359 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:48:44.777341    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4f6a368c-bc67-4bda-9fd6-554d4e20d681-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-5cd44d6886-4v9nb\" (UID: \"4f6a368c-bc67-4bda-9fd6-554d4e20d681\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5cd44d6886-4v9nb"
Sep 16 17:48:44.779101 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:48:44.779084    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4f6a368c-bc67-4bda-9fd6-554d4e20d681-proxy-tls\") pod \"isvc-canary-create-v2-predictor-5cd44d6886-4v9nb\" (UID: \"4f6a368c-bc67-4bda-9fd6-554d4e20d681\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5cd44d6886-4v9nb"
Sep 16 17:48:44.917223 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:48:44.917173    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5cd44d6886-4v9nb"
Sep 16 17:48:44.938801 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:48:44.938769    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-npzlr"
Sep 16 17:48:45.205091 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:48:45.205051    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5cd44d6886-4v9nb"]
Sep 16 17:48:45.208621 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:48:45.208589    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f6a368c_bc67_4bda_9fd6_554d4e20d681.slice/crio-9d320af86db2d647dae91fb54a05a5fa0f86d6ef95b3438b532ca98f50a95e54 WatchSource:0}: Error finding container 9d320af86db2d647dae91fb54a05a5fa0f86d6ef95b3438b532ca98f50a95e54: Status 404 returned error can't find the container with id 9d320af86db2d647dae91fb54a05a5fa0f86d6ef95b3438b532ca98f50a95e54
Sep 16 17:48:45.228207 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:48:45.228183    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-npzlr"]
Sep 16 17:48:45.230645 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:48:45.230621    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedfe795b_2f83_445b_95de_bfb096759517.slice/crio-36d000eaf4895a1faa59538e5511d9aa2373a2f280e38a666692e802c1fe1701 WatchSource:0}: Error finding container 36d000eaf4895a1faa59538e5511d9aa2373a2f280e38a666692e802c1fe1701: Status 404 returned error can't find the container with id 36d000eaf4895a1faa59538e5511d9aa2373a2f280e38a666692e802c1fe1701
Sep 16 17:48:45.814236 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:48:45.814193    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-npzlr" event={"ID":"edfe795b-2f83-445b-95de-bfb096759517","Type":"ContainerStarted","Data":"521eca23071f823b0143d7c2d8ff26143ccd1370074348192a6fbc53874c9b65"}
Sep 16 17:48:45.814435 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:48:45.814282    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-npzlr" event={"ID":"edfe795b-2f83-445b-95de-bfb096759517","Type":"ContainerStarted","Data":"36d000eaf4895a1faa59538e5511d9aa2373a2f280e38a666692e802c1fe1701"}
Sep 16 17:48:45.815928 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:48:45.815902    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5cd44d6886-4v9nb" event={"ID":"4f6a368c-bc67-4bda-9fd6-554d4e20d681","Type":"ContainerStarted","Data":"699ff9821a5263e51427c9c5312782f13ed92ac3e4cc73b833eded17a1984233"}
Sep 16 17:48:45.816074 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:48:45.815951    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5cd44d6886-4v9nb" event={"ID":"4f6a368c-bc67-4bda-9fd6-554d4e20d681","Type":"ContainerStarted","Data":"9d320af86db2d647dae91fb54a05a5fa0f86d6ef95b3438b532ca98f50a95e54"}
Sep 16 17:48:49.838407 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:48:49.838376    2579 generic.go:356] "Generic (PLEG): container finished" podID="edfe795b-2f83-445b-95de-bfb096759517" containerID="521eca23071f823b0143d7c2d8ff26143ccd1370074348192a6fbc53874c9b65" exitCode=0
Sep 16 17:48:49.838868 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:48:49.838453    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-npzlr" event={"ID":"edfe795b-2f83-445b-95de-bfb096759517","Type":"ContainerDied","Data":"521eca23071f823b0143d7c2d8ff26143ccd1370074348192a6fbc53874c9b65"}
Sep 16 17:48:49.839957 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:48:49.839937    2579 generic.go:356] "Generic (PLEG): container finished" podID="4f6a368c-bc67-4bda-9fd6-554d4e20d681" containerID="699ff9821a5263e51427c9c5312782f13ed92ac3e4cc73b833eded17a1984233" exitCode=0
Sep 16 17:48:49.840066 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:48:49.840008    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5cd44d6886-4v9nb" event={"ID":"4f6a368c-bc67-4bda-9fd6-554d4e20d681","Type":"ContainerDied","Data":"699ff9821a5263e51427c9c5312782f13ed92ac3e4cc73b833eded17a1984233"}
Sep 16 17:48:50.846820 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:48:50.846786    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-npzlr" event={"ID":"edfe795b-2f83-445b-95de-bfb096759517","Type":"ContainerStarted","Data":"499c0f8007bd855c3495989821cc9e406a0c82346dabbf350b21d2bfa22ce1e8"}
Sep 16 17:48:50.847246 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:48:50.846879    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-npzlr" event={"ID":"edfe795b-2f83-445b-95de-bfb096759517","Type":"ContainerStarted","Data":"ce2a5b8dfc5214753da6345cede371cf59c6923478da0f7c1ee3361cdbdd20b9"}
Sep 16 17:48:50.847246 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:48:50.847195    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-npzlr"
Sep 16 17:48:50.848863 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:48:50.848829    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5cd44d6886-4v9nb" event={"ID":"4f6a368c-bc67-4bda-9fd6-554d4e20d681","Type":"ContainerStarted","Data":"cfada6411621064740e3e66b06f28fd2ac8aff0ac9adc8e14a1f7735089ec416"}
Sep 16 17:48:50.849005 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:48:50.848898    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5cd44d6886-4v9nb" event={"ID":"4f6a368c-bc67-4bda-9fd6-554d4e20d681","Type":"ContainerStarted","Data":"ad6b581f9b3c28a8bd63c47baecdcfc334cc94baa7d447695ce77c944e13f993"}
Sep 16 17:48:50.849105 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:48:50.849087    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5cd44d6886-4v9nb"
Sep 16 17:48:50.879273 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:48:50.879221    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-npzlr" podStartSLOduration=3.884381608 podStartE2EDuration="7.879205128s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.99482352s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:48:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:48:50.874433006 +0000 UTC m=+1443.344431141" watchObservedRunningTime="2026-09-16 17:48:50.879205128 +0000 UTC m=+1443.349203263"
Sep 16 17:48:50.908660 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:48:50.908603    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5cd44d6886-4v9nb" podStartSLOduration=4.021459159 podStartE2EDuration="7.908585121s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.887125962s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:48:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:48:50.906261377 +0000 UTC m=+1443.376259513" watchObservedRunningTime="2026-09-16 17:48:50.908585121 +0000 UTC m=+1443.378583257"
Sep 16 17:48:51.854173 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:48:51.854144    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5cd44d6886-4v9nb"
Sep 16 17:48:51.854609 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:48:51.854212    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-npzlr"
Sep 16 17:48:51.855071 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:48:51.855044    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-npzlr" podUID="edfe795b-2f83-445b-95de-bfb096759517" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Sep 16 17:48:51.855147 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:48:51.855119    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5cd44d6886-4v9nb" podUID="4f6a368c-bc67-4bda-9fd6-554d4e20d681" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Sep 16 17:48:52.858328 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:48:52.858293    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5cd44d6886-4v9nb" podUID="4f6a368c-bc67-4bda-9fd6-554d4e20d681" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Sep 16 17:48:52.858723 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:48:52.858296    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-npzlr" podUID="edfe795b-2f83-445b-95de-bfb096759517" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Sep 16 17:48:57.862584 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:48:57.862547    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-npzlr"
Sep 16 17:48:57.863167 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:48:57.863143    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5cd44d6886-4v9nb"
Sep 16 17:48:57.863633 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:48:57.863602    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-npzlr" podUID="edfe795b-2f83-445b-95de-bfb096759517" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Sep 16 17:48:57.863741 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:48:57.863716    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5cd44d6886-4v9nb" podUID="4f6a368c-bc67-4bda-9fd6-554d4e20d681" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Sep 16 17:49:07.863980 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:49:07.863938    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5cd44d6886-4v9nb" podUID="4f6a368c-bc67-4bda-9fd6-554d4e20d681" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Sep 16 17:49:07.864373 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:49:07.864000    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-npzlr" podUID="edfe795b-2f83-445b-95de-bfb096759517" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Sep 16 17:49:17.864028 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:49:17.863977    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-npzlr" podUID="edfe795b-2f83-445b-95de-bfb096759517" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Sep 16 17:49:17.864606 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:49:17.863977    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5cd44d6886-4v9nb" podUID="4f6a368c-bc67-4bda-9fd6-554d4e20d681" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Sep 16 17:49:27.863831 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:49:27.863788    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-npzlr" podUID="edfe795b-2f83-445b-95de-bfb096759517" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Sep 16 17:49:27.864350 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:49:27.863800    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5cd44d6886-4v9nb" podUID="4f6a368c-bc67-4bda-9fd6-554d4e20d681" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Sep 16 17:49:37.864447 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:49:37.864401    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-npzlr" podUID="edfe795b-2f83-445b-95de-bfb096759517" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Sep 16 17:49:37.864447 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:49:37.864429    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5cd44d6886-4v9nb" podUID="4f6a368c-bc67-4bda-9fd6-554d4e20d681" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Sep 16 17:49:47.863967 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:49:47.863922    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5cd44d6886-4v9nb" podUID="4f6a368c-bc67-4bda-9fd6-554d4e20d681" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Sep 16 17:49:47.864456 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:49:47.863975    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-npzlr" podUID="edfe795b-2f83-445b-95de-bfb096759517" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Sep 16 17:49:57.864287 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:49:57.864237    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-npzlr" podUID="edfe795b-2f83-445b-95de-bfb096759517" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Sep 16 17:49:57.864977 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:49:57.864237    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5cd44d6886-4v9nb" podUID="4f6a368c-bc67-4bda-9fd6-554d4e20d681" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Sep 16 17:50:00.094294 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:50:00.094252    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-npzlr" podUID="edfe795b-2f83-445b-95de-bfb096759517" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Sep 16 17:50:07.864541 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:50:07.864509    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5cd44d6886-4v9nb"
Sep 16 17:50:07.997968 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:50:07.997935    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-npzlr"]
Sep 16 17:50:07.998210 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:50:07.998163    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-npzlr" podUID="edfe795b-2f83-445b-95de-bfb096759517" containerName="kserve-container" containerID="cri-o://ce2a5b8dfc5214753da6345cede371cf59c6923478da0f7c1ee3361cdbdd20b9" gracePeriod=30
Sep 16 17:50:07.998284 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:50:07.998230    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-npzlr" podUID="edfe795b-2f83-445b-95de-bfb096759517" containerName="kube-rbac-proxy" containerID="cri-o://499c0f8007bd855c3495989821cc9e406a0c82346dabbf350b21d2bfa22ce1e8" gracePeriod=30
Sep 16 17:50:07.999045 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:50:07.999021    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-npzlr"
Sep 16 17:50:08.217382 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:50:08.217347    2579 generic.go:356] "Generic (PLEG): container finished" podID="edfe795b-2f83-445b-95de-bfb096759517" containerID="499c0f8007bd855c3495989821cc9e406a0c82346dabbf350b21d2bfa22ce1e8" exitCode=2
Sep 16 17:50:08.217559 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:50:08.217424    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-npzlr" event={"ID":"edfe795b-2f83-445b-95de-bfb096759517","Type":"ContainerDied","Data":"499c0f8007bd855c3495989821cc9e406a0c82346dabbf350b21d2bfa22ce1e8"}
Sep 16 17:50:12.858276 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:50:12.858230    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-npzlr" podUID="edfe795b-2f83-445b-95de-bfb096759517" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.45:8643/healthz\": dial tcp 10.134.0.45:8643: connect: connection refused"
Sep 16 17:50:12.948363 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:50:12.948335    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-npzlr"
Sep 16 17:50:13.072386 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:50:13.072337    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/edfe795b-2f83-445b-95de-bfb096759517-cabundle-cert\") pod \"edfe795b-2f83-445b-95de-bfb096759517\" (UID: \"edfe795b-2f83-445b-95de-bfb096759517\") "
Sep 16 17:50:13.072386 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:50:13.072386    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/edfe795b-2f83-445b-95de-bfb096759517-kube-rbac-proxy-sar-config\") pod \"edfe795b-2f83-445b-95de-bfb096759517\" (UID: \"edfe795b-2f83-445b-95de-bfb096759517\") "
Sep 16 17:50:13.072608 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:50:13.072433    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zmch2\" (UniqueName: \"kubernetes.io/projected/edfe795b-2f83-445b-95de-bfb096759517-kube-api-access-zmch2\") pod \"edfe795b-2f83-445b-95de-bfb096759517\" (UID: \"edfe795b-2f83-445b-95de-bfb096759517\") "
Sep 16 17:50:13.072608 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:50:13.072451    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/edfe795b-2f83-445b-95de-bfb096759517-kserve-provision-location\") pod \"edfe795b-2f83-445b-95de-bfb096759517\" (UID: \"edfe795b-2f83-445b-95de-bfb096759517\") "
Sep 16 17:50:13.072608 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:50:13.072524    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/edfe795b-2f83-445b-95de-bfb096759517-proxy-tls\") pod \"edfe795b-2f83-445b-95de-bfb096759517\" (UID: \"edfe795b-2f83-445b-95de-bfb096759517\") "
Sep 16 17:50:13.072883 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:50:13.072826    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/edfe795b-2f83-445b-95de-bfb096759517-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "edfe795b-2f83-445b-95de-bfb096759517" (UID: "edfe795b-2f83-445b-95de-bfb096759517"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 17:50:13.073035 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:50:13.072868    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edfe795b-2f83-445b-95de-bfb096759517-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "edfe795b-2f83-445b-95de-bfb096759517" (UID: "edfe795b-2f83-445b-95de-bfb096759517"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:50:13.073035 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:50:13.072939    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/edfe795b-2f83-445b-95de-bfb096759517-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "edfe795b-2f83-445b-95de-bfb096759517" (UID: "edfe795b-2f83-445b-95de-bfb096759517"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 17:50:13.074762 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:50:13.074734    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/edfe795b-2f83-445b-95de-bfb096759517-kube-api-access-zmch2" (OuterVolumeSpecName: "kube-api-access-zmch2") pod "edfe795b-2f83-445b-95de-bfb096759517" (UID: "edfe795b-2f83-445b-95de-bfb096759517"). InnerVolumeSpecName "kube-api-access-zmch2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 17:50:13.074890 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:50:13.074786    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/edfe795b-2f83-445b-95de-bfb096759517-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "edfe795b-2f83-445b-95de-bfb096759517" (UID: "edfe795b-2f83-445b-95de-bfb096759517"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 17:50:13.166139 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:50:13.166106    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5cd44d6886-4v9nb"]
Sep 16 17:50:13.166433 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:50:13.166408    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5cd44d6886-4v9nb" podUID="4f6a368c-bc67-4bda-9fd6-554d4e20d681" containerName="kserve-container" containerID="cri-o://ad6b581f9b3c28a8bd63c47baecdcfc334cc94baa7d447695ce77c944e13f993" gracePeriod=30
Sep 16 17:50:13.166529 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:50:13.166426    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5cd44d6886-4v9nb" podUID="4f6a368c-bc67-4bda-9fd6-554d4e20d681" containerName="kube-rbac-proxy" containerID="cri-o://cfada6411621064740e3e66b06f28fd2ac8aff0ac9adc8e14a1f7735089ec416" gracePeriod=30
Sep 16 17:50:13.173876 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:50:13.173837    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zmch2\" (UniqueName: \"kubernetes.io/projected/edfe795b-2f83-445b-95de-bfb096759517-kube-api-access-zmch2\") on node \"ip-10-0-138-73.ec2.internal\" DevicePath \"\""
Sep 16 17:50:13.173980 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:50:13.173879    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/edfe795b-2f83-445b-95de-bfb096759517-kserve-provision-location\") on node \"ip-10-0-138-73.ec2.internal\" DevicePath \"\""
Sep 16 17:50:13.173980 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:50:13.173895    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/edfe795b-2f83-445b-95de-bfb096759517-proxy-tls\") on node \"ip-10-0-138-73.ec2.internal\" DevicePath \"\""
Sep 16 17:50:13.173980 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:50:13.173909    2579 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/edfe795b-2f83-445b-95de-bfb096759517-cabundle-cert\") on node \"ip-10-0-138-73.ec2.internal\" DevicePath \"\""
Sep 16 17:50:13.173980 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:50:13.173921    2579 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/edfe795b-2f83-445b-95de-bfb096759517-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-73.ec2.internal\" DevicePath \"\""
Sep 16 17:50:13.243108 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:50:13.243069    2579 generic.go:356] "Generic (PLEG): container finished" podID="edfe795b-2f83-445b-95de-bfb096759517" containerID="ce2a5b8dfc5214753da6345cede371cf59c6923478da0f7c1ee3361cdbdd20b9" exitCode=0
Sep 16 17:50:13.243270 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:50:13.243148    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-npzlr" event={"ID":"edfe795b-2f83-445b-95de-bfb096759517","Type":"ContainerDied","Data":"ce2a5b8dfc5214753da6345cede371cf59c6923478da0f7c1ee3361cdbdd20b9"}
Sep 16 17:50:13.243270 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:50:13.243199    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-npzlr" event={"ID":"edfe795b-2f83-445b-95de-bfb096759517","Type":"ContainerDied","Data":"36d000eaf4895a1faa59538e5511d9aa2373a2f280e38a666692e802c1fe1701"}
Sep 16 17:50:13.243270 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:50:13.243226    2579 scope.go:117] "RemoveContainer" containerID="499c0f8007bd855c3495989821cc9e406a0c82346dabbf350b21d2bfa22ce1e8"
Sep 16 17:50:13.243270 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:50:13.243150    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-npzlr"
Sep 16 17:50:13.266275 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:50:13.266244    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-npzlr"]
Sep 16 17:50:13.271103 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:50:13.271077    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-npzlr"]
Sep 16 17:50:13.277414 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:50:13.277386    2579 scope.go:117] "RemoveContainer" containerID="ce2a5b8dfc5214753da6345cede371cf59c6923478da0f7c1ee3361cdbdd20b9"
Sep 16 17:50:13.285573 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:50:13.285555    2579 scope.go:117] "RemoveContainer" containerID="521eca23071f823b0143d7c2d8ff26143ccd1370074348192a6fbc53874c9b65"
Sep 16 17:50:13.293802 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:50:13.293780    2579 scope.go:117] "RemoveContainer" containerID="499c0f8007bd855c3495989821cc9e406a0c82346dabbf350b21d2bfa22ce1e8"
Sep 16 17:50:13.294115 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:50:13.294097    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"499c0f8007bd855c3495989821cc9e406a0c82346dabbf350b21d2bfa22ce1e8\": container with ID starting with 499c0f8007bd855c3495989821cc9e406a0c82346dabbf350b21d2bfa22ce1e8 not found: ID does not exist" containerID="499c0f8007bd855c3495989821cc9e406a0c82346dabbf350b21d2bfa22ce1e8"
Sep 16 17:50:13.294184 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:50:13.294124    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"499c0f8007bd855c3495989821cc9e406a0c82346dabbf350b21d2bfa22ce1e8"} err="failed to get container status \"499c0f8007bd855c3495989821cc9e406a0c82346dabbf350b21d2bfa22ce1e8\": rpc error: code = NotFound desc = could not find container \"499c0f8007bd855c3495989821cc9e406a0c82346dabbf350b21d2bfa22ce1e8\": container with ID starting with 499c0f8007bd855c3495989821cc9e406a0c82346dabbf350b21d2bfa22ce1e8 not found: ID does not exist"
Sep 16 17:50:13.294184 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:50:13.294142    2579 scope.go:117] "RemoveContainer" containerID="ce2a5b8dfc5214753da6345cede371cf59c6923478da0f7c1ee3361cdbdd20b9"
Sep 16 17:50:13.294431 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:50:13.294407    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce2a5b8dfc5214753da6345cede371cf59c6923478da0f7c1ee3361cdbdd20b9\": container with ID starting with ce2a5b8dfc5214753da6345cede371cf59c6923478da0f7c1ee3361cdbdd20b9 not found: ID does not exist" containerID="ce2a5b8dfc5214753da6345cede371cf59c6923478da0f7c1ee3361cdbdd20b9"
Sep 16 17:50:13.294472 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:50:13.294441    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce2a5b8dfc5214753da6345cede371cf59c6923478da0f7c1ee3361cdbdd20b9"} err="failed to get container status \"ce2a5b8dfc5214753da6345cede371cf59c6923478da0f7c1ee3361cdbdd20b9\": rpc error: code = NotFound desc = could not find container \"ce2a5b8dfc5214753da6345cede371cf59c6923478da0f7c1ee3361cdbdd20b9\": container with ID starting with ce2a5b8dfc5214753da6345cede371cf59c6923478da0f7c1ee3361cdbdd20b9 not found: ID does not exist"
Sep 16 17:50:13.294472 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:50:13.294466    2579 scope.go:117] "RemoveContainer" containerID="521eca23071f823b0143d7c2d8ff26143ccd1370074348192a6fbc53874c9b65"
Sep 16 17:50:13.294723 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:50:13.294707    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"521eca23071f823b0143d7c2d8ff26143ccd1370074348192a6fbc53874c9b65\": container with ID starting with 521eca23071f823b0143d7c2d8ff26143ccd1370074348192a6fbc53874c9b65 not found: ID does not exist" containerID="521eca23071f823b0143d7c2d8ff26143ccd1370074348192a6fbc53874c9b65"
Sep 16 17:50:13.294765 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:50:13.294728    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"521eca23071f823b0143d7c2d8ff26143ccd1370074348192a6fbc53874c9b65"} err="failed to get container status \"521eca23071f823b0143d7c2d8ff26143ccd1370074348192a6fbc53874c9b65\": rpc error: code = NotFound desc = could not find container \"521eca23071f823b0143d7c2d8ff26143ccd1370074348192a6fbc53874c9b65\": container with ID starting with 521eca23071f823b0143d7c2d8ff26143ccd1370074348192a6fbc53874c9b65 not found: ID does not exist"
Sep 16 17:50:14.099749 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:50:14.099715    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="edfe795b-2f83-445b-95de-bfb096759517" path="/var/lib/kubelet/pods/edfe795b-2f83-445b-95de-bfb096759517/volumes"
Sep 16 17:50:14.251422 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:50:14.251393    2579 generic.go:356] "Generic (PLEG): container finished" podID="4f6a368c-bc67-4bda-9fd6-554d4e20d681" containerID="cfada6411621064740e3e66b06f28fd2ac8aff0ac9adc8e14a1f7735089ec416" exitCode=2
Sep 16 17:50:14.251576 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:50:14.251428    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5cd44d6886-4v9nb" event={"ID":"4f6a368c-bc67-4bda-9fd6-554d4e20d681","Type":"ContainerDied","Data":"cfada6411621064740e3e66b06f28fd2ac8aff0ac9adc8e14a1f7735089ec416"}
Sep 16 17:50:17.859210 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:50:17.859168    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5cd44d6886-4v9nb" podUID="4f6a368c-bc67-4bda-9fd6-554d4e20d681" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.44:8643/healthz\": dial tcp 10.134.0.44:8643: connect: connection refused"
Sep 16 17:50:17.863999 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:50:17.863965    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5cd44d6886-4v9nb" podUID="4f6a368c-bc67-4bda-9fd6-554d4e20d681" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Sep 16 17:50:18.025768 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:50:18.025742    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5cd44d6886-4v9nb"
Sep 16 17:50:18.114671 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:50:18.114597    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4f6a368c-bc67-4bda-9fd6-554d4e20d681-proxy-tls\") pod \"4f6a368c-bc67-4bda-9fd6-554d4e20d681\" (UID: \"4f6a368c-bc67-4bda-9fd6-554d4e20d681\") "
Sep 16 17:50:18.114671 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:50:18.114661    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4f6a368c-bc67-4bda-9fd6-554d4e20d681-cabundle-cert\") pod \"4f6a368c-bc67-4bda-9fd6-554d4e20d681\" (UID: \"4f6a368c-bc67-4bda-9fd6-554d4e20d681\") "
Sep 16 17:50:18.114905 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:50:18.114700    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4f6a368c-bc67-4bda-9fd6-554d4e20d681-kserve-provision-location\") pod \"4f6a368c-bc67-4bda-9fd6-554d4e20d681\" (UID: \"4f6a368c-bc67-4bda-9fd6-554d4e20d681\") "
Sep 16 17:50:18.114905 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:50:18.114731    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4vgzg\" (UniqueName: \"kubernetes.io/projected/4f6a368c-bc67-4bda-9fd6-554d4e20d681-kube-api-access-4vgzg\") pod \"4f6a368c-bc67-4bda-9fd6-554d4e20d681\" (UID: \"4f6a368c-bc67-4bda-9fd6-554d4e20d681\") "
Sep 16 17:50:18.114905 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:50:18.114766    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4f6a368c-bc67-4bda-9fd6-554d4e20d681-kube-rbac-proxy-sar-config\") pod \"4f6a368c-bc67-4bda-9fd6-554d4e20d681\" (UID: \"4f6a368c-bc67-4bda-9fd6-554d4e20d681\") "
Sep 16 17:50:18.115077 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:50:18.115025    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f6a368c-bc67-4bda-9fd6-554d4e20d681-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4f6a368c-bc67-4bda-9fd6-554d4e20d681" (UID: "4f6a368c-bc67-4bda-9fd6-554d4e20d681"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:50:18.115135 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:50:18.115088    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f6a368c-bc67-4bda-9fd6-554d4e20d681-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "4f6a368c-bc67-4bda-9fd6-554d4e20d681" (UID: "4f6a368c-bc67-4bda-9fd6-554d4e20d681"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 17:50:18.115179 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:50:18.115130    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f6a368c-bc67-4bda-9fd6-554d4e20d681-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "4f6a368c-bc67-4bda-9fd6-554d4e20d681" (UID: "4f6a368c-bc67-4bda-9fd6-554d4e20d681"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 17:50:18.116729 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:50:18.116706    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f6a368c-bc67-4bda-9fd6-554d4e20d681-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4f6a368c-bc67-4bda-9fd6-554d4e20d681" (UID: "4f6a368c-bc67-4bda-9fd6-554d4e20d681"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 17:50:18.116840 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:50:18.116824    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f6a368c-bc67-4bda-9fd6-554d4e20d681-kube-api-access-4vgzg" (OuterVolumeSpecName: "kube-api-access-4vgzg") pod "4f6a368c-bc67-4bda-9fd6-554d4e20d681" (UID: "4f6a368c-bc67-4bda-9fd6-554d4e20d681"). InnerVolumeSpecName "kube-api-access-4vgzg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 17:50:18.216100 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:50:18.216059    2579 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4f6a368c-bc67-4bda-9fd6-554d4e20d681-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-73.ec2.internal\" DevicePath \"\""
Sep 16 17:50:18.216100 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:50:18.216093    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4f6a368c-bc67-4bda-9fd6-554d4e20d681-proxy-tls\") on node \"ip-10-0-138-73.ec2.internal\" DevicePath \"\""
Sep 16 17:50:18.216100 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:50:18.216109    2579 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4f6a368c-bc67-4bda-9fd6-554d4e20d681-cabundle-cert\") on node \"ip-10-0-138-73.ec2.internal\" DevicePath \"\""
Sep 16 17:50:18.216339 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:50:18.216117    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4f6a368c-bc67-4bda-9fd6-554d4e20d681-kserve-provision-location\") on node \"ip-10-0-138-73.ec2.internal\" DevicePath \"\""
Sep 16 17:50:18.216339 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:50:18.216126    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4vgzg\" (UniqueName: \"kubernetes.io/projected/4f6a368c-bc67-4bda-9fd6-554d4e20d681-kube-api-access-4vgzg\") on node \"ip-10-0-138-73.ec2.internal\" DevicePath \"\""
Sep 16 17:50:18.274376 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:50:18.274340    2579 generic.go:356] "Generic (PLEG): container finished" podID="4f6a368c-bc67-4bda-9fd6-554d4e20d681" containerID="ad6b581f9b3c28a8bd63c47baecdcfc334cc94baa7d447695ce77c944e13f993" exitCode=0
Sep 16 17:50:18.274559 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:50:18.274398    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5cd44d6886-4v9nb"
Sep 16 17:50:18.274559 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:50:18.274430    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5cd44d6886-4v9nb" event={"ID":"4f6a368c-bc67-4bda-9fd6-554d4e20d681","Type":"ContainerDied","Data":"ad6b581f9b3c28a8bd63c47baecdcfc334cc94baa7d447695ce77c944e13f993"}
Sep 16 17:50:18.274559 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:50:18.274495    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5cd44d6886-4v9nb" event={"ID":"4f6a368c-bc67-4bda-9fd6-554d4e20d681","Type":"ContainerDied","Data":"9d320af86db2d647dae91fb54a05a5fa0f86d6ef95b3438b532ca98f50a95e54"}
Sep 16 17:50:18.274559 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:50:18.274522    2579 scope.go:117] "RemoveContainer" containerID="cfada6411621064740e3e66b06f28fd2ac8aff0ac9adc8e14a1f7735089ec416"
Sep 16 17:50:18.284378 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:50:18.284349    2579 scope.go:117] "RemoveContainer" containerID="ad6b581f9b3c28a8bd63c47baecdcfc334cc94baa7d447695ce77c944e13f993"
Sep 16 17:50:18.293622 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:50:18.293597    2579 scope.go:117] "RemoveContainer" containerID="699ff9821a5263e51427c9c5312782f13ed92ac3e4cc73b833eded17a1984233"
Sep 16 17:50:18.302901 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:50:18.302878    2579 scope.go:117] "RemoveContainer" containerID="cfada6411621064740e3e66b06f28fd2ac8aff0ac9adc8e14a1f7735089ec416"
Sep 16 17:50:18.303217 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:50:18.303196    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cfada6411621064740e3e66b06f28fd2ac8aff0ac9adc8e14a1f7735089ec416\": container with ID starting with cfada6411621064740e3e66b06f28fd2ac8aff0ac9adc8e14a1f7735089ec416 not found: ID does not exist" containerID="cfada6411621064740e3e66b06f28fd2ac8aff0ac9adc8e14a1f7735089ec416"
Sep 16 17:50:18.303297 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:50:18.303227    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cfada6411621064740e3e66b06f28fd2ac8aff0ac9adc8e14a1f7735089ec416"} err="failed to get container status \"cfada6411621064740e3e66b06f28fd2ac8aff0ac9adc8e14a1f7735089ec416\": rpc error: code = NotFound desc = could not find container \"cfada6411621064740e3e66b06f28fd2ac8aff0ac9adc8e14a1f7735089ec416\": container with ID starting with cfada6411621064740e3e66b06f28fd2ac8aff0ac9adc8e14a1f7735089ec416 not found: ID does not exist"
Sep 16 17:50:18.303297 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:50:18.303245    2579 scope.go:117] "RemoveContainer" containerID="ad6b581f9b3c28a8bd63c47baecdcfc334cc94baa7d447695ce77c944e13f993"
Sep 16 17:50:18.303535 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:50:18.303518    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad6b581f9b3c28a8bd63c47baecdcfc334cc94baa7d447695ce77c944e13f993\": container with ID starting with ad6b581f9b3c28a8bd63c47baecdcfc334cc94baa7d447695ce77c944e13f993 not found: ID does not exist" containerID="ad6b581f9b3c28a8bd63c47baecdcfc334cc94baa7d447695ce77c944e13f993"
Sep 16 17:50:18.303581 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:50:18.303539    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad6b581f9b3c28a8bd63c47baecdcfc334cc94baa7d447695ce77c944e13f993"} err="failed to get container status \"ad6b581f9b3c28a8bd63c47baecdcfc334cc94baa7d447695ce77c944e13f993\": rpc error: code = NotFound desc = could not find container \"ad6b581f9b3c28a8bd63c47baecdcfc334cc94baa7d447695ce77c944e13f993\": container with ID starting with ad6b581f9b3c28a8bd63c47baecdcfc334cc94baa7d447695ce77c944e13f993 not found: ID does not exist"
Sep 16 17:50:18.303581 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:50:18.303552    2579 scope.go:117] "RemoveContainer" containerID="699ff9821a5263e51427c9c5312782f13ed92ac3e4cc73b833eded17a1984233"
Sep 16 17:50:18.303828 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:50:18.303807    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"699ff9821a5263e51427c9c5312782f13ed92ac3e4cc73b833eded17a1984233\": container with ID starting with 699ff9821a5263e51427c9c5312782f13ed92ac3e4cc73b833eded17a1984233 not found: ID does not exist" containerID="699ff9821a5263e51427c9c5312782f13ed92ac3e4cc73b833eded17a1984233"
Sep 16 17:50:18.303945 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:50:18.303831    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"699ff9821a5263e51427c9c5312782f13ed92ac3e4cc73b833eded17a1984233"} err="failed to get container status \"699ff9821a5263e51427c9c5312782f13ed92ac3e4cc73b833eded17a1984233\": rpc error: code = NotFound desc = could not find container \"699ff9821a5263e51427c9c5312782f13ed92ac3e4cc73b833eded17a1984233\": container with ID starting with 699ff9821a5263e51427c9c5312782f13ed92ac3e4cc73b833eded17a1984233 not found: ID does not exist"
Sep 16 17:50:18.331226 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:50:18.331192    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5cd44d6886-4v9nb"]
Sep 16 17:50:18.338006 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:50:18.337982    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5cd44d6886-4v9nb"]
Sep 16 17:50:18.608291 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:50:18.608261    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-vkjss"]
Sep 16 17:50:18.608622 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:50:18.608611    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f6a368c-bc67-4bda-9fd6-554d4e20d681" containerName="kube-rbac-proxy"
Sep 16 17:50:18.608666 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:50:18.608624    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f6a368c-bc67-4bda-9fd6-554d4e20d681" containerName="kube-rbac-proxy"
Sep 16 17:50:18.608666 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:50:18.608639    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f6a368c-bc67-4bda-9fd6-554d4e20d681" containerName="kserve-container"
Sep 16 17:50:18.608666 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:50:18.608645    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f6a368c-bc67-4bda-9fd6-554d4e20d681" containerName="kserve-container"
Sep 16 17:50:18.608666 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:50:18.608653    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="edfe795b-2f83-445b-95de-bfb096759517" containerName="storage-initializer"
Sep 16 17:50:18.608666 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:50:18.608659    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="edfe795b-2f83-445b-95de-bfb096759517" containerName="storage-initializer"
Sep 16 17:50:18.608666 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:50:18.608665    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="edfe795b-2f83-445b-95de-bfb096759517" containerName="kube-rbac-proxy"
Sep 16 17:50:18.608874 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:50:18.608671    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="edfe795b-2f83-445b-95de-bfb096759517" containerName="kube-rbac-proxy"
Sep 16 17:50:18.608874 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:50:18.608679    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="edfe795b-2f83-445b-95de-bfb096759517" containerName="kserve-container"
Sep 16 17:50:18.608874 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:50:18.608684    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="edfe795b-2f83-445b-95de-bfb096759517" containerName="kserve-container"
Sep 16 17:50:18.608874 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:50:18.608697    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f6a368c-bc67-4bda-9fd6-554d4e20d681" containerName="storage-initializer"
Sep 16 17:50:18.608874 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:50:18.608702    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f6a368c-bc67-4bda-9fd6-554d4e20d681" containerName="storage-initializer"
Sep 16 17:50:18.608874 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:50:18.608759    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="edfe795b-2f83-445b-95de-bfb096759517" containerName="kube-rbac-proxy"
Sep 16 17:50:18.608874 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:50:18.608766    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="edfe795b-2f83-445b-95de-bfb096759517" containerName="kserve-container"
Sep 16 17:50:18.608874 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:50:18.608773    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="4f6a368c-bc67-4bda-9fd6-554d4e20d681" containerName="kserve-container"
Sep 16 17:50:18.608874 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:50:18.608781    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="4f6a368c-bc67-4bda-9fd6-554d4e20d681" containerName="kube-rbac-proxy"
Sep 16 17:50:18.611774 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:50:18.611750    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-vkjss"
Sep 16 17:50:18.615675 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:50:18.615648    2579 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"default-dockercfg-9bc9t\" is forbidden: User \"system:node:ip-10-0-138-73.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"kserve-ci-e2e-test\": no relationship found between node 'ip-10-0-138-73.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-9bc9t\"" type="*v1.Secret"
Sep 16 17:50:18.616948 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:50:18.616914    2579 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"openshift-service-ca.crt\" is forbidden: User \"system:node:ip-10-0-138-73.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"kserve-ci-e2e-test\": no relationship found between node 'ip-10-0-138-73.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap"
Sep 16 17:50:18.616948 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:50:18.616915    2579 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"seaweedfs-s3-creds\" is forbidden: User \"system:node:ip-10-0-138-73.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"kserve-ci-e2e-test\": no relationship found between node 'ip-10-0-138-73.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\"" type="*v1.Secret"
Sep 16 17:50:18.617124 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:50:18.617030    2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"isvc-canary-promote-predictor-65ff4fc5f5-vkjss\" is forbidden: User \"system:node:ip-10-0-138-73.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kserve-ci-e2e-test\": no relationship found between node 'ip-10-0-138-73.ec2.internal' and this object" podUID="2928c230-1840-489d-9a62-759097efe734" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-vkjss"
Sep 16 17:50:18.617124 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:50:18.617070    2579 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"kube-root-ca.crt\" is forbidden: User \"system:node:ip-10-0-138-73.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"kserve-ci-e2e-test\": no relationship found between node 'ip-10-0-138-73.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap"
Sep 16 17:50:18.617124 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:50:18.617087    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-predictor-serving-cert\""
Sep 16 17:50:18.617304 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:50:18.617164    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 17:50:18.617304 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:50:18.617070    2579 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"isvc-canary-promote-kube-rbac-proxy-sar-config\" is forbidden: User \"system:node:ip-10-0-138-73.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"kserve-ci-e2e-test\": no relationship found between node 'ip-10-0-138-73.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-kube-rbac-proxy-sar-config\"" type="*v1.ConfigMap"
Sep 16 17:50:18.619271 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:50:18.619241    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2928c230-1840-489d-9a62-759097efe734-proxy-tls\") pod \"isvc-canary-promote-predictor-65ff4fc5f5-vkjss\" (UID: \"2928c230-1840-489d-9a62-759097efe734\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-vkjss"
Sep 16 17:50:18.619366 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:50:18.619280    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2928c230-1840-489d-9a62-759097efe734-cabundle-cert\") pod \"isvc-canary-promote-predictor-65ff4fc5f5-vkjss\" (UID: \"2928c230-1840-489d-9a62-759097efe734\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-vkjss"
Sep 16 17:50:18.619366 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:50:18.619350    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2928c230-1840-489d-9a62-759097efe734-kserve-provision-location\") pod \"isvc-canary-promote-predictor-65ff4fc5f5-vkjss\" (UID: \"2928c230-1840-489d-9a62-759097efe734\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-vkjss"
Sep 16 17:50:18.619478 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:50:18.619440    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2928c230-1840-489d-9a62-759097efe734-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-65ff4fc5f5-vkjss\" (UID: \"2928c230-1840-489d-9a62-759097efe734\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-vkjss"
Sep 16 17:50:18.619478 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:50:18.619469    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxpvd\" (UniqueName: \"kubernetes.io/projected/2928c230-1840-489d-9a62-759097efe734-kube-api-access-nxpvd\") pod \"isvc-canary-promote-predictor-65ff4fc5f5-vkjss\" (UID: \"2928c230-1840-489d-9a62-759097efe734\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-vkjss"
Sep 16 17:50:18.634807 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:50:18.634780    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-vkjss"]
Sep 16 17:50:18.720679 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:50:18.720641    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2928c230-1840-489d-9a62-759097efe734-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-65ff4fc5f5-vkjss\" (UID: \"2928c230-1840-489d-9a62-759097efe734\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-vkjss"
Sep 16 17:50:18.720679 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:50:18.720677    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxpvd\" (UniqueName: \"kubernetes.io/projected/2928c230-1840-489d-9a62-759097efe734-kube-api-access-nxpvd\") pod \"isvc-canary-promote-predictor-65ff4fc5f5-vkjss\" (UID: \"2928c230-1840-489d-9a62-759097efe734\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-vkjss"
Sep 16 17:50:18.720934 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:50:18.720726    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2928c230-1840-489d-9a62-759097efe734-proxy-tls\") pod \"isvc-canary-promote-predictor-65ff4fc5f5-vkjss\" (UID: \"2928c230-1840-489d-9a62-759097efe734\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-vkjss"
Sep 16 17:50:18.720934 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:50:18.720743    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2928c230-1840-489d-9a62-759097efe734-cabundle-cert\") pod \"isvc-canary-promote-predictor-65ff4fc5f5-vkjss\" (UID: \"2928c230-1840-489d-9a62-759097efe734\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-vkjss"
Sep 16 17:50:18.720934 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:50:18.720771    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2928c230-1840-489d-9a62-759097efe734-kserve-provision-location\") pod \"isvc-canary-promote-predictor-65ff4fc5f5-vkjss\" (UID: \"2928c230-1840-489d-9a62-759097efe734\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-vkjss"
Sep 16 17:50:18.720934 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:50:18.720878    2579 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-promote-predictor-serving-cert: secret "isvc-canary-promote-predictor-serving-cert" not found
Sep 16 17:50:18.721202 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:50:18.720955    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2928c230-1840-489d-9a62-759097efe734-proxy-tls podName:2928c230-1840-489d-9a62-759097efe734 nodeName:}" failed. No retries permitted until 2026-09-16 17:50:19.22093762 +0000 UTC m=+1531.690935751 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/2928c230-1840-489d-9a62-759097efe734-proxy-tls") pod "isvc-canary-promote-predictor-65ff4fc5f5-vkjss" (UID: "2928c230-1840-489d-9a62-759097efe734") : secret "isvc-canary-promote-predictor-serving-cert" not found
Sep 16 17:50:18.721257 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:50:18.721207    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2928c230-1840-489d-9a62-759097efe734-kserve-provision-location\") pod \"isvc-canary-promote-predictor-65ff4fc5f5-vkjss\" (UID: \"2928c230-1840-489d-9a62-759097efe734\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-vkjss"
Sep 16 17:50:18.721465 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:50:18.721447    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2928c230-1840-489d-9a62-759097efe734-cabundle-cert\") pod \"isvc-canary-promote-predictor-65ff4fc5f5-vkjss\" (UID: \"2928c230-1840-489d-9a62-759097efe734\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-vkjss"
Sep 16 17:50:19.225644 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:50:19.225585    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2928c230-1840-489d-9a62-759097efe734-proxy-tls\") pod \"isvc-canary-promote-predictor-65ff4fc5f5-vkjss\" (UID: \"2928c230-1840-489d-9a62-759097efe734\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-vkjss"
Sep 16 17:50:19.228129 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:50:19.228106    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2928c230-1840-489d-9a62-759097efe734-proxy-tls\") pod \"isvc-canary-promote-predictor-65ff4fc5f5-vkjss\" (UID: \"2928c230-1840-489d-9a62-759097efe734\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-vkjss"
Sep 16 17:50:19.475151 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:50:19.475119    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 16 17:50:19.720958 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:50:19.720922    2579 configmap.go:193] Couldn't get configMap kserve-ci-e2e-test/isvc-canary-promote-kube-rbac-proxy-sar-config: failed to sync configmap cache: timed out waiting for the condition
Sep 16 17:50:19.721110 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:50:19.721001    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2928c230-1840-489d-9a62-759097efe734-kube-rbac-proxy-sar-config podName:2928c230-1840-489d-9a62-759097efe734 nodeName:}" failed. No retries permitted until 2026-09-16 17:50:20.220982981 +0000 UTC m=+1532.690981098 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-sar-config" (UniqueName: "kubernetes.io/configmap/2928c230-1840-489d-9a62-759097efe734-kube-rbac-proxy-sar-config") pod "isvc-canary-promote-predictor-65ff4fc5f5-vkjss" (UID: "2928c230-1840-489d-9a62-759097efe734") : failed to sync configmap cache: timed out waiting for the condition
Sep 16 17:50:19.999752 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:50:19.999669    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 16 17:50:20.035775 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:50:20.035747    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-9bc9t\""
Sep 16 17:50:20.100451 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:50:20.100416    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4f6a368c-bc67-4bda-9fd6-554d4e20d681" path="/var/lib/kubelet/pods/4f6a368c-bc67-4bda-9fd6-554d4e20d681/volumes"
Sep 16 17:50:20.193500 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:50:20.193466    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-kube-rbac-proxy-sar-config\""
Sep 16 17:50:20.196893 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:50:20.196865    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 16 17:50:20.206467 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:50:20.206442    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxpvd\" (UniqueName: \"kubernetes.io/projected/2928c230-1840-489d-9a62-759097efe734-kube-api-access-nxpvd\") pod \"isvc-canary-promote-predictor-65ff4fc5f5-vkjss\" (UID: \"2928c230-1840-489d-9a62-759097efe734\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-vkjss"
Sep 16 17:50:20.235302 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:50:20.235265    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2928c230-1840-489d-9a62-759097efe734-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-65ff4fc5f5-vkjss\" (UID: \"2928c230-1840-489d-9a62-759097efe734\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-vkjss"
Sep 16 17:50:20.236048 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:50:20.236023    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2928c230-1840-489d-9a62-759097efe734-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-65ff4fc5f5-vkjss\" (UID: \"2928c230-1840-489d-9a62-759097efe734\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-vkjss"
Sep 16 17:50:20.422121 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:50:20.422074    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-vkjss"
Sep 16 17:50:20.699065 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:50:20.699039    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-vkjss"]
Sep 16 17:50:20.700502 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:50:20.700475    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2928c230_1840_489d_9a62_759097efe734.slice/crio-2ff2e6c914c8b0bc17baffbb88db3cdccc1e1acc6e4e1fa37cc7f0c9370a7694 WatchSource:0}: Error finding container 2ff2e6c914c8b0bc17baffbb88db3cdccc1e1acc6e4e1fa37cc7f0c9370a7694: Status 404 returned error can't find the container with id 2ff2e6c914c8b0bc17baffbb88db3cdccc1e1acc6e4e1fa37cc7f0c9370a7694
Sep 16 17:50:21.291297 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:50:21.291264    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-vkjss" event={"ID":"2928c230-1840-489d-9a62-759097efe734","Type":"ContainerStarted","Data":"a465ac3b4561d5e57039b21ebefb86cbcbbb936754964d52b0d34cf4a5fb289b"}
Sep 16 17:50:21.291665 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:50:21.291320    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-vkjss" event={"ID":"2928c230-1840-489d-9a62-759097efe734","Type":"ContainerStarted","Data":"2ff2e6c914c8b0bc17baffbb88db3cdccc1e1acc6e4e1fa37cc7f0c9370a7694"}
Sep 16 17:50:25.311638 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:50:25.311605    2579 generic.go:356] "Generic (PLEG): container finished" podID="2928c230-1840-489d-9a62-759097efe734" containerID="a465ac3b4561d5e57039b21ebefb86cbcbbb936754964d52b0d34cf4a5fb289b" exitCode=0
Sep 16 17:50:25.312015 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:50:25.311686    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-vkjss" event={"ID":"2928c230-1840-489d-9a62-759097efe734","Type":"ContainerDied","Data":"a465ac3b4561d5e57039b21ebefb86cbcbbb936754964d52b0d34cf4a5fb289b"}
Sep 16 17:50:26.319498 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:50:26.319465    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-vkjss" event={"ID":"2928c230-1840-489d-9a62-759097efe734","Type":"ContainerStarted","Data":"6c8285966657421ae1bb14625f0b7ae315bee47aadc1687c0f1468fb798fde33"}
Sep 16 17:50:26.319967 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:50:26.319529    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-vkjss" event={"ID":"2928c230-1840-489d-9a62-759097efe734","Type":"ContainerStarted","Data":"4070c98f63989a18ded8b012dfc1012612103073a54f117bc18062ac0aa95c89"}
Sep 16 17:50:26.319967 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:50:26.319793    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-vkjss"
Sep 16 17:50:26.319967 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:50:26.319950    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-vkjss"
Sep 16 17:50:26.321228 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:50:26.321205    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-vkjss" podUID="2928c230-1840-489d-9a62-759097efe734" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Sep 16 17:50:26.342390 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:50:26.342340    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-vkjss" podStartSLOduration=4.606944107 podStartE2EDuration="8.342323617s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.73537951s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:50:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:50:26.340544661 +0000 UTC m=+1538.810542798" watchObservedRunningTime="2026-09-16 17:50:26.342323617 +0000 UTC m=+1538.812321752"
Sep 16 17:50:27.324218 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:50:27.324176    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-vkjss" podUID="2928c230-1840-489d-9a62-759097efe734" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Sep 16 17:50:32.328811 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:50:32.328779    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-vkjss"
Sep 16 17:50:32.331264 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:50:32.329288    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-vkjss" podUID="2928c230-1840-489d-9a62-759097efe734" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Sep 16 17:50:42.329699 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:50:42.329658    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-vkjss" podUID="2928c230-1840-489d-9a62-759097efe734" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Sep 16 17:50:52.330152 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:50:52.330113    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-vkjss" podUID="2928c230-1840-489d-9a62-759097efe734" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Sep 16 17:51:02.329289 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:51:02.329246    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-vkjss" podUID="2928c230-1840-489d-9a62-759097efe734" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Sep 16 17:51:12.329284 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:51:12.329236    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-vkjss" podUID="2928c230-1840-489d-9a62-759097efe734" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Sep 16 17:51:22.329794 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:51:22.329750    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-vkjss" podUID="2928c230-1840-489d-9a62-759097efe734" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Sep 16 17:51:32.329643 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:51:32.329600    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-vkjss" podUID="2928c230-1840-489d-9a62-759097efe734" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Sep 16 17:51:42.330684 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:51:42.330606    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-vkjss"
Sep 16 17:51:48.333218 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:51:48.333165    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-vkjss"]
Sep 16 17:51:48.333679 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:51:48.333417    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-vkjss" podUID="2928c230-1840-489d-9a62-759097efe734" containerName="kserve-container" containerID="cri-o://4070c98f63989a18ded8b012dfc1012612103073a54f117bc18062ac0aa95c89" gracePeriod=30
Sep 16 17:51:48.333679 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:51:48.333494    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-vkjss" podUID="2928c230-1840-489d-9a62-759097efe734" containerName="kube-rbac-proxy" containerID="cri-o://6c8285966657421ae1bb14625f0b7ae315bee47aadc1687c0f1468fb798fde33" gracePeriod=30
Sep 16 17:51:48.451403 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:51:48.451381    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-qcfjg"]
Sep 16 17:51:48.455109 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:51:48.455089    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-qcfjg"
Sep 16 17:51:48.459541 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:51:48.459515    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-v2-predictor-serving-cert\""
Sep 16 17:51:48.471718 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:51:48.471683    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-qcfjg"]
Sep 16 17:51:48.617562 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:51:48.617480    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cdf1c8d9-1736-4ae0-ad71-ae966d09a6ce-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-b8964cf76-qcfjg\" (UID: \"cdf1c8d9-1736-4ae0-ad71-ae966d09a6ce\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-qcfjg"
Sep 16 17:51:48.617562 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:51:48.617530    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cdf1c8d9-1736-4ae0-ad71-ae966d09a6ce-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-b8964cf76-qcfjg\" (UID: \"cdf1c8d9-1736-4ae0-ad71-ae966d09a6ce\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-qcfjg"
Sep 16 17:51:48.617562 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:51:48.617555    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t58v2\" (UniqueName: \"kubernetes.io/projected/cdf1c8d9-1736-4ae0-ad71-ae966d09a6ce-kube-api-access-t58v2\") pod \"isvc-canary-promote-v2-predictor-b8964cf76-qcfjg\" (UID: \"cdf1c8d9-1736-4ae0-ad71-ae966d09a6ce\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-qcfjg"
Sep 16 17:51:48.617789 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:51:48.617665    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cdf1c8d9-1736-4ae0-ad71-ae966d09a6ce-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-b8964cf76-qcfjg\" (UID: \"cdf1c8d9-1736-4ae0-ad71-ae966d09a6ce\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-qcfjg"
Sep 16 17:51:48.617789 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:51:48.617732    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cdf1c8d9-1736-4ae0-ad71-ae966d09a6ce-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-b8964cf76-qcfjg\" (UID: \"cdf1c8d9-1736-4ae0-ad71-ae966d09a6ce\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-qcfjg"
Sep 16 17:51:48.718213 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:51:48.718175    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cdf1c8d9-1736-4ae0-ad71-ae966d09a6ce-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-b8964cf76-qcfjg\" (UID: \"cdf1c8d9-1736-4ae0-ad71-ae966d09a6ce\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-qcfjg"
Sep 16 17:51:48.718387 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:51:48.718247    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cdf1c8d9-1736-4ae0-ad71-ae966d09a6ce-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-b8964cf76-qcfjg\" (UID: \"cdf1c8d9-1736-4ae0-ad71-ae966d09a6ce\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-qcfjg"
Sep 16 17:51:48.718387 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:51:48.718284    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cdf1c8d9-1736-4ae0-ad71-ae966d09a6ce-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-b8964cf76-qcfjg\" (UID: \"cdf1c8d9-1736-4ae0-ad71-ae966d09a6ce\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-qcfjg"
Sep 16 17:51:48.718387 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:51:48.718347    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cdf1c8d9-1736-4ae0-ad71-ae966d09a6ce-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-b8964cf76-qcfjg\" (UID: \"cdf1c8d9-1736-4ae0-ad71-ae966d09a6ce\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-qcfjg"
Sep 16 17:51:48.718387 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:51:48.718380    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t58v2\" (UniqueName: \"kubernetes.io/projected/cdf1c8d9-1736-4ae0-ad71-ae966d09a6ce-kube-api-access-t58v2\") pod \"isvc-canary-promote-v2-predictor-b8964cf76-qcfjg\" (UID: \"cdf1c8d9-1736-4ae0-ad71-ae966d09a6ce\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-qcfjg"
Sep 16 17:51:48.718839 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:51:48.718814    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cdf1c8d9-1736-4ae0-ad71-ae966d09a6ce-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-b8964cf76-qcfjg\" (UID: \"cdf1c8d9-1736-4ae0-ad71-ae966d09a6ce\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-qcfjg"
Sep 16 17:51:48.719085 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:51:48.719065    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cdf1c8d9-1736-4ae0-ad71-ae966d09a6ce-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-b8964cf76-qcfjg\" (UID: \"cdf1c8d9-1736-4ae0-ad71-ae966d09a6ce\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-qcfjg"
Sep 16 17:51:48.719432 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:51:48.719408    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cdf1c8d9-1736-4ae0-ad71-ae966d09a6ce-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-b8964cf76-qcfjg\" (UID: \"cdf1c8d9-1736-4ae0-ad71-ae966d09a6ce\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-qcfjg"
Sep 16 17:51:48.720167 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:51:48.720143    2579 generic.go:356] "Generic (PLEG): container finished" podID="2928c230-1840-489d-9a62-759097efe734" containerID="6c8285966657421ae1bb14625f0b7ae315bee47aadc1687c0f1468fb798fde33" exitCode=2
Sep 16 17:51:48.720239 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:51:48.720192    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-vkjss" event={"ID":"2928c230-1840-489d-9a62-759097efe734","Type":"ContainerDied","Data":"6c8285966657421ae1bb14625f0b7ae315bee47aadc1687c0f1468fb798fde33"}
Sep 16 17:51:48.720915 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:51:48.720898    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cdf1c8d9-1736-4ae0-ad71-ae966d09a6ce-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-b8964cf76-qcfjg\" (UID: \"cdf1c8d9-1736-4ae0-ad71-ae966d09a6ce\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-qcfjg"
Sep 16 17:51:48.728740 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:51:48.728722    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t58v2\" (UniqueName: \"kubernetes.io/projected/cdf1c8d9-1736-4ae0-ad71-ae966d09a6ce-kube-api-access-t58v2\") pod \"isvc-canary-promote-v2-predictor-b8964cf76-qcfjg\" (UID: \"cdf1c8d9-1736-4ae0-ad71-ae966d09a6ce\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-qcfjg"
Sep 16 17:51:48.765154 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:51:48.765126    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-qcfjg"
Sep 16 17:51:49.054295 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:51:49.054267    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-qcfjg"]
Sep 16 17:51:49.056996 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:51:49.056963    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdf1c8d9_1736_4ae0_ad71_ae966d09a6ce.slice/crio-e97211cfab1ce24aefee0035b8f520a64d97fc35b0522825bafb37591bd607f3 WatchSource:0}: Error finding container e97211cfab1ce24aefee0035b8f520a64d97fc35b0522825bafb37591bd607f3: Status 404 returned error can't find the container with id e97211cfab1ce24aefee0035b8f520a64d97fc35b0522825bafb37591bd607f3
Sep 16 17:51:49.727028 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:51:49.726992    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-qcfjg" event={"ID":"cdf1c8d9-1736-4ae0-ad71-ae966d09a6ce","Type":"ContainerStarted","Data":"b0c10c8014ef1c32e61b27131dae367dac5c2984bed7505e251dc2c9704590e6"}
Sep 16 17:51:49.727472 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:51:49.727058    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-qcfjg" event={"ID":"cdf1c8d9-1736-4ae0-ad71-ae966d09a6ce","Type":"ContainerStarted","Data":"e97211cfab1ce24aefee0035b8f520a64d97fc35b0522825bafb37591bd607f3"}
Sep 16 17:51:52.324559 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:51:52.324509    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-vkjss" podUID="2928c230-1840-489d-9a62-759097efe734" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.46:8643/healthz\": dial tcp 10.134.0.46:8643: connect: connection refused"
Sep 16 17:51:52.329293 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:51:52.329265    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-vkjss" podUID="2928c230-1840-489d-9a62-759097efe734" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Sep 16 17:51:53.282919 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:51:53.282896    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-vkjss"
Sep 16 17:51:53.362772 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:51:53.362736    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2928c230-1840-489d-9a62-759097efe734-kube-rbac-proxy-sar-config\") pod \"2928c230-1840-489d-9a62-759097efe734\" (UID: \"2928c230-1840-489d-9a62-759097efe734\") "
Sep 16 17:51:53.363253 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:51:53.362790    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2928c230-1840-489d-9a62-759097efe734-proxy-tls\") pod \"2928c230-1840-489d-9a62-759097efe734\" (UID: \"2928c230-1840-489d-9a62-759097efe734\") "
Sep 16 17:51:53.363253 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:51:53.362819    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2928c230-1840-489d-9a62-759097efe734-kserve-provision-location\") pod \"2928c230-1840-489d-9a62-759097efe734\" (UID: \"2928c230-1840-489d-9a62-759097efe734\") "
Sep 16 17:51:53.363253 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:51:53.362874    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nxpvd\" (UniqueName: \"kubernetes.io/projected/2928c230-1840-489d-9a62-759097efe734-kube-api-access-nxpvd\") pod \"2928c230-1840-489d-9a62-759097efe734\" (UID: \"2928c230-1840-489d-9a62-759097efe734\") "
Sep 16 17:51:53.363253 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:51:53.362947    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2928c230-1840-489d-9a62-759097efe734-cabundle-cert\") pod \"2928c230-1840-489d-9a62-759097efe734\" (UID: \"2928c230-1840-489d-9a62-759097efe734\") "
Sep 16 17:51:53.363253 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:51:53.363140    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2928c230-1840-489d-9a62-759097efe734-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "2928c230-1840-489d-9a62-759097efe734" (UID: "2928c230-1840-489d-9a62-759097efe734"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 17:51:53.363253 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:51:53.363156    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2928c230-1840-489d-9a62-759097efe734-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2928c230-1840-489d-9a62-759097efe734" (UID: "2928c230-1840-489d-9a62-759097efe734"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:51:53.363253 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:51:53.363215    2579 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2928c230-1840-489d-9a62-759097efe734-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-73.ec2.internal\" DevicePath \"\""
Sep 16 17:51:53.363519 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:51:53.363338    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2928c230-1840-489d-9a62-759097efe734-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "2928c230-1840-489d-9a62-759097efe734" (UID: "2928c230-1840-489d-9a62-759097efe734"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 17:51:53.365002 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:51:53.364980    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2928c230-1840-489d-9a62-759097efe734-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "2928c230-1840-489d-9a62-759097efe734" (UID: "2928c230-1840-489d-9a62-759097efe734"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 17:51:53.365074 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:51:53.365041    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2928c230-1840-489d-9a62-759097efe734-kube-api-access-nxpvd" (OuterVolumeSpecName: "kube-api-access-nxpvd") pod "2928c230-1840-489d-9a62-759097efe734" (UID: "2928c230-1840-489d-9a62-759097efe734"). InnerVolumeSpecName "kube-api-access-nxpvd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 17:51:53.464557 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:51:53.464458    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2928c230-1840-489d-9a62-759097efe734-proxy-tls\") on node \"ip-10-0-138-73.ec2.internal\" DevicePath \"\""
Sep 16 17:51:53.464557 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:51:53.464501    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2928c230-1840-489d-9a62-759097efe734-kserve-provision-location\") on node \"ip-10-0-138-73.ec2.internal\" DevicePath \"\""
Sep 16 17:51:53.464557 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:51:53.464511    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nxpvd\" (UniqueName: \"kubernetes.io/projected/2928c230-1840-489d-9a62-759097efe734-kube-api-access-nxpvd\") on node \"ip-10-0-138-73.ec2.internal\" DevicePath \"\""
Sep 16 17:51:53.464557 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:51:53.464520    2579 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2928c230-1840-489d-9a62-759097efe734-cabundle-cert\") on node \"ip-10-0-138-73.ec2.internal\" DevicePath \"\""
Sep 16 17:51:53.747148 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:51:53.747060    2579 generic.go:356] "Generic (PLEG): container finished" podID="cdf1c8d9-1736-4ae0-ad71-ae966d09a6ce" containerID="b0c10c8014ef1c32e61b27131dae367dac5c2984bed7505e251dc2c9704590e6" exitCode=0
Sep 16 17:51:53.747304 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:51:53.747138    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-qcfjg" event={"ID":"cdf1c8d9-1736-4ae0-ad71-ae966d09a6ce","Type":"ContainerDied","Data":"b0c10c8014ef1c32e61b27131dae367dac5c2984bed7505e251dc2c9704590e6"}
Sep 16 17:51:53.749045 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:51:53.749021    2579 generic.go:356] "Generic (PLEG): container finished" podID="2928c230-1840-489d-9a62-759097efe734" containerID="4070c98f63989a18ded8b012dfc1012612103073a54f117bc18062ac0aa95c89" exitCode=0
Sep 16 17:51:53.749147 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:51:53.749075    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-vkjss"
Sep 16 17:51:53.749147 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:51:53.749095    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-vkjss" event={"ID":"2928c230-1840-489d-9a62-759097efe734","Type":"ContainerDied","Data":"4070c98f63989a18ded8b012dfc1012612103073a54f117bc18062ac0aa95c89"}
Sep 16 17:51:53.749222 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:51:53.749149    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-vkjss" event={"ID":"2928c230-1840-489d-9a62-759097efe734","Type":"ContainerDied","Data":"2ff2e6c914c8b0bc17baffbb88db3cdccc1e1acc6e4e1fa37cc7f0c9370a7694"}
Sep 16 17:51:53.749222 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:51:53.749174    2579 scope.go:117] "RemoveContainer" containerID="6c8285966657421ae1bb14625f0b7ae315bee47aadc1687c0f1468fb798fde33"
Sep 16 17:51:53.761110 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:51:53.761094    2579 scope.go:117] "RemoveContainer" containerID="4070c98f63989a18ded8b012dfc1012612103073a54f117bc18062ac0aa95c89"
Sep 16 17:51:53.770075 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:51:53.770053    2579 scope.go:117] "RemoveContainer" containerID="a465ac3b4561d5e57039b21ebefb86cbcbbb936754964d52b0d34cf4a5fb289b"
Sep 16 17:51:53.786019 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:51:53.785998    2579 scope.go:117] "RemoveContainer" containerID="6c8285966657421ae1bb14625f0b7ae315bee47aadc1687c0f1468fb798fde33"
Sep 16 17:51:53.786352 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:51:53.786333    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c8285966657421ae1bb14625f0b7ae315bee47aadc1687c0f1468fb798fde33\": container with ID starting with 6c8285966657421ae1bb14625f0b7ae315bee47aadc1687c0f1468fb798fde33 not found: ID does not exist" containerID="6c8285966657421ae1bb14625f0b7ae315bee47aadc1687c0f1468fb798fde33"
Sep 16 17:51:53.786431 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:51:53.786362    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c8285966657421ae1bb14625f0b7ae315bee47aadc1687c0f1468fb798fde33"} err="failed to get container status \"6c8285966657421ae1bb14625f0b7ae315bee47aadc1687c0f1468fb798fde33\": rpc error: code = NotFound desc = could not find container \"6c8285966657421ae1bb14625f0b7ae315bee47aadc1687c0f1468fb798fde33\": container with ID starting with 6c8285966657421ae1bb14625f0b7ae315bee47aadc1687c0f1468fb798fde33 not found: ID does not exist"
Sep 16 17:51:53.786431 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:51:53.786381    2579 scope.go:117] "RemoveContainer" containerID="4070c98f63989a18ded8b012dfc1012612103073a54f117bc18062ac0aa95c89"
Sep 16 17:51:53.786625 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:51:53.786606    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4070c98f63989a18ded8b012dfc1012612103073a54f117bc18062ac0aa95c89\": container with ID starting with 4070c98f63989a18ded8b012dfc1012612103073a54f117bc18062ac0aa95c89 not found: ID does not exist" containerID="4070c98f63989a18ded8b012dfc1012612103073a54f117bc18062ac0aa95c89"
Sep 16 17:51:53.786696 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:51:53.786637    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4070c98f63989a18ded8b012dfc1012612103073a54f117bc18062ac0aa95c89"} err="failed to get container status \"4070c98f63989a18ded8b012dfc1012612103073a54f117bc18062ac0aa95c89\": rpc error: code = NotFound desc = could not find container \"4070c98f63989a18ded8b012dfc1012612103073a54f117bc18062ac0aa95c89\": container with ID starting with 4070c98f63989a18ded8b012dfc1012612103073a54f117bc18062ac0aa95c89 not found: ID does not exist"
Sep 16 17:51:53.786696 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:51:53.786660    2579 scope.go:117] "RemoveContainer" containerID="a465ac3b4561d5e57039b21ebefb86cbcbbb936754964d52b0d34cf4a5fb289b"
Sep 16 17:51:53.786987 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:51:53.786963    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a465ac3b4561d5e57039b21ebefb86cbcbbb936754964d52b0d34cf4a5fb289b\": container with ID starting with a465ac3b4561d5e57039b21ebefb86cbcbbb936754964d52b0d34cf4a5fb289b not found: ID does not exist" containerID="a465ac3b4561d5e57039b21ebefb86cbcbbb936754964d52b0d34cf4a5fb289b"
Sep 16 17:51:53.787043 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:51:53.786985    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a465ac3b4561d5e57039b21ebefb86cbcbbb936754964d52b0d34cf4a5fb289b"} err="failed to get container status \"a465ac3b4561d5e57039b21ebefb86cbcbbb936754964d52b0d34cf4a5fb289b\": rpc error: code = NotFound desc = could not find container \"a465ac3b4561d5e57039b21ebefb86cbcbbb936754964d52b0d34cf4a5fb289b\": container with ID starting with a465ac3b4561d5e57039b21ebefb86cbcbbb936754964d52b0d34cf4a5fb289b not found: ID does not exist"
Sep 16 17:51:53.790883 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:51:53.790862    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-vkjss"]
Sep 16 17:51:53.796435 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:51:53.796415    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-vkjss"]
Sep 16 17:51:54.100448 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:51:54.100417    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2928c230-1840-489d-9a62-759097efe734" path="/var/lib/kubelet/pods/2928c230-1840-489d-9a62-759097efe734/volumes"
Sep 16 17:51:54.756592 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:51:54.756559    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-qcfjg" event={"ID":"cdf1c8d9-1736-4ae0-ad71-ae966d09a6ce","Type":"ContainerStarted","Data":"e4f756e9db5bff3f94813262ad0a2289f29e91302c724e04aa4342d6b4540bbd"}
Sep 16 17:51:54.757019 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:51:54.756615    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-qcfjg" event={"ID":"cdf1c8d9-1736-4ae0-ad71-ae966d09a6ce","Type":"ContainerStarted","Data":"3367e50e47f261902be2e9bab286f77e4fcfe77eb31b6f844e865d348e960b11"}
Sep 16 17:51:54.757019 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:51:54.756883    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-qcfjg"
Sep 16 17:51:54.757135 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:51:54.757041    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-qcfjg"
Sep 16 17:51:54.758387 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:51:54.758362    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-qcfjg" podUID="cdf1c8d9-1736-4ae0-ad71-ae966d09a6ce" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Sep 16 17:51:54.795328 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:51:54.795265    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-qcfjg" podStartSLOduration=3.049560201 podStartE2EDuration="6.795244927s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.745684726s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:51:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:51:54.794693321 +0000 UTC m=+1627.264691473" watchObservedRunningTime="2026-09-16 17:51:54.795244927 +0000 UTC m=+1627.265243079"
Sep 16 17:51:55.762211 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:51:55.762171    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-qcfjg" podUID="cdf1c8d9-1736-4ae0-ad71-ae966d09a6ce" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Sep 16 17:52:00.767040 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:52:00.767003    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-qcfjg"
Sep 16 17:52:00.767619 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:52:00.767586    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-qcfjg" podUID="cdf1c8d9-1736-4ae0-ad71-ae966d09a6ce" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Sep 16 17:52:10.767465 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:52:10.767427    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-qcfjg" podUID="cdf1c8d9-1736-4ae0-ad71-ae966d09a6ce" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Sep 16 17:52:20.767958 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:52:20.767920    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-qcfjg" podUID="cdf1c8d9-1736-4ae0-ad71-ae966d09a6ce" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Sep 16 17:52:30.767471 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:52:30.767426    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-qcfjg" podUID="cdf1c8d9-1736-4ae0-ad71-ae966d09a6ce" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Sep 16 17:52:40.767690 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:52:40.767646    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-qcfjg" podUID="cdf1c8d9-1736-4ae0-ad71-ae966d09a6ce" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Sep 16 17:52:50.767758 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:52:50.767705    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-qcfjg" podUID="cdf1c8d9-1736-4ae0-ad71-ae966d09a6ce" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Sep 16 17:53:00.767725 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:53:00.767683    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-qcfjg" podUID="cdf1c8d9-1736-4ae0-ad71-ae966d09a6ce" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Sep 16 17:53:08.100272 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:53:08.100226    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-qcfjg" podUID="cdf1c8d9-1736-4ae0-ad71-ae966d09a6ce" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Sep 16 17:53:18.101036 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:53:18.100954    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-qcfjg"
Sep 16 17:53:18.640825 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:53:18.640794    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-qcfjg"]
Sep 16 17:53:18.641057 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:53:18.641014    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-qcfjg" podUID="cdf1c8d9-1736-4ae0-ad71-ae966d09a6ce" containerName="kserve-container" containerID="cri-o://3367e50e47f261902be2e9bab286f77e4fcfe77eb31b6f844e865d348e960b11" gracePeriod=30
Sep 16 17:53:18.641126 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:53:18.641053    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-qcfjg" podUID="cdf1c8d9-1736-4ae0-ad71-ae966d09a6ce" containerName="kube-rbac-proxy" containerID="cri-o://e4f756e9db5bff3f94813262ad0a2289f29e91302c724e04aa4342d6b4540bbd" gracePeriod=30
Sep 16 17:53:19.164361 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:53:19.164330    2579 generic.go:356] "Generic (PLEG): container finished" podID="cdf1c8d9-1736-4ae0-ad71-ae966d09a6ce" containerID="e4f756e9db5bff3f94813262ad0a2289f29e91302c724e04aa4342d6b4540bbd" exitCode=2
Sep 16 17:53:19.164936 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:53:19.164398    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-qcfjg" event={"ID":"cdf1c8d9-1736-4ae0-ad71-ae966d09a6ce","Type":"ContainerDied","Data":"e4f756e9db5bff3f94813262ad0a2289f29e91302c724e04aa4342d6b4540bbd"}
Sep 16 17:53:20.762602 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:53:20.762561    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-qcfjg" podUID="cdf1c8d9-1736-4ae0-ad71-ae966d09a6ce" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.47:8643/healthz\": dial tcp 10.134.0.47:8643: connect: connection refused"
Sep 16 17:53:23.489692 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:53:23.489667    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-qcfjg"
Sep 16 17:53:23.658923 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:53:23.658889    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cdf1c8d9-1736-4ae0-ad71-ae966d09a6ce-kube-rbac-proxy-sar-config\") pod \"cdf1c8d9-1736-4ae0-ad71-ae966d09a6ce\" (UID: \"cdf1c8d9-1736-4ae0-ad71-ae966d09a6ce\") "
Sep 16 17:53:23.659105 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:53:23.658959    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cdf1c8d9-1736-4ae0-ad71-ae966d09a6ce-proxy-tls\") pod \"cdf1c8d9-1736-4ae0-ad71-ae966d09a6ce\" (UID: \"cdf1c8d9-1736-4ae0-ad71-ae966d09a6ce\") "
Sep 16 17:53:23.659105 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:53:23.659002    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cdf1c8d9-1736-4ae0-ad71-ae966d09a6ce-kserve-provision-location\") pod \"cdf1c8d9-1736-4ae0-ad71-ae966d09a6ce\" (UID: \"cdf1c8d9-1736-4ae0-ad71-ae966d09a6ce\") "
Sep 16 17:53:23.659105 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:53:23.659026    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t58v2\" (UniqueName: \"kubernetes.io/projected/cdf1c8d9-1736-4ae0-ad71-ae966d09a6ce-kube-api-access-t58v2\") pod \"cdf1c8d9-1736-4ae0-ad71-ae966d09a6ce\" (UID: \"cdf1c8d9-1736-4ae0-ad71-ae966d09a6ce\") "
Sep 16 17:53:23.659268 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:53:23.659110    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cdf1c8d9-1736-4ae0-ad71-ae966d09a6ce-cabundle-cert\") pod \"cdf1c8d9-1736-4ae0-ad71-ae966d09a6ce\" (UID: \"cdf1c8d9-1736-4ae0-ad71-ae966d09a6ce\") "
Sep 16 17:53:23.659398 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:53:23.659371    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdf1c8d9-1736-4ae0-ad71-ae966d09a6ce-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "cdf1c8d9-1736-4ae0-ad71-ae966d09a6ce" (UID: "cdf1c8d9-1736-4ae0-ad71-ae966d09a6ce"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:53:23.659398 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:53:23.659386    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cdf1c8d9-1736-4ae0-ad71-ae966d09a6ce-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "cdf1c8d9-1736-4ae0-ad71-ae966d09a6ce" (UID: "cdf1c8d9-1736-4ae0-ad71-ae966d09a6ce"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 17:53:23.659480 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:53:23.659403    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cdf1c8d9-1736-4ae0-ad71-ae966d09a6ce-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "cdf1c8d9-1736-4ae0-ad71-ae966d09a6ce" (UID: "cdf1c8d9-1736-4ae0-ad71-ae966d09a6ce"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 17:53:23.661215 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:53:23.661190    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cdf1c8d9-1736-4ae0-ad71-ae966d09a6ce-kube-api-access-t58v2" (OuterVolumeSpecName: "kube-api-access-t58v2") pod "cdf1c8d9-1736-4ae0-ad71-ae966d09a6ce" (UID: "cdf1c8d9-1736-4ae0-ad71-ae966d09a6ce"). InnerVolumeSpecName "kube-api-access-t58v2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 17:53:23.661318 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:53:23.661259    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cdf1c8d9-1736-4ae0-ad71-ae966d09a6ce-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "cdf1c8d9-1736-4ae0-ad71-ae966d09a6ce" (UID: "cdf1c8d9-1736-4ae0-ad71-ae966d09a6ce"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 17:53:23.759970 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:53:23.759933    2579 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cdf1c8d9-1736-4ae0-ad71-ae966d09a6ce-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-73.ec2.internal\" DevicePath \"\""
Sep 16 17:53:23.759970 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:53:23.759962    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cdf1c8d9-1736-4ae0-ad71-ae966d09a6ce-proxy-tls\") on node \"ip-10-0-138-73.ec2.internal\" DevicePath \"\""
Sep 16 17:53:23.759970 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:53:23.759973    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cdf1c8d9-1736-4ae0-ad71-ae966d09a6ce-kserve-provision-location\") on node \"ip-10-0-138-73.ec2.internal\" DevicePath \"\""
Sep 16 17:53:23.759970 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:53:23.759983    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t58v2\" (UniqueName: \"kubernetes.io/projected/cdf1c8d9-1736-4ae0-ad71-ae966d09a6ce-kube-api-access-t58v2\") on node \"ip-10-0-138-73.ec2.internal\" DevicePath \"\""
Sep 16 17:53:23.760234 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:53:23.759991    2579 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cdf1c8d9-1736-4ae0-ad71-ae966d09a6ce-cabundle-cert\") on node \"ip-10-0-138-73.ec2.internal\" DevicePath \"\""
Sep 16 17:53:24.189112 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:53:24.189076    2579 generic.go:356] "Generic (PLEG): container finished" podID="cdf1c8d9-1736-4ae0-ad71-ae966d09a6ce" containerID="3367e50e47f261902be2e9bab286f77e4fcfe77eb31b6f844e865d348e960b11" exitCode=0
Sep 16 17:53:24.189277 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:53:24.189156    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-qcfjg"
Sep 16 17:53:24.189277 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:53:24.189160    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-qcfjg" event={"ID":"cdf1c8d9-1736-4ae0-ad71-ae966d09a6ce","Type":"ContainerDied","Data":"3367e50e47f261902be2e9bab286f77e4fcfe77eb31b6f844e865d348e960b11"}
Sep 16 17:53:24.189277 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:53:24.189229    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-qcfjg" event={"ID":"cdf1c8d9-1736-4ae0-ad71-ae966d09a6ce","Type":"ContainerDied","Data":"e97211cfab1ce24aefee0035b8f520a64d97fc35b0522825bafb37591bd607f3"}
Sep 16 17:53:24.189277 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:53:24.189259    2579 scope.go:117] "RemoveContainer" containerID="e4f756e9db5bff3f94813262ad0a2289f29e91302c724e04aa4342d6b4540bbd"
Sep 16 17:53:24.200188 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:53:24.200170    2579 scope.go:117] "RemoveContainer" containerID="3367e50e47f261902be2e9bab286f77e4fcfe77eb31b6f844e865d348e960b11"
Sep 16 17:53:24.207839 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:53:24.207736    2579 scope.go:117] "RemoveContainer" containerID="b0c10c8014ef1c32e61b27131dae367dac5c2984bed7505e251dc2c9704590e6"
Sep 16 17:53:24.210179 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:53:24.210157    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-qcfjg"]
Sep 16 17:53:24.216583 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:53:24.216563    2579 scope.go:117] "RemoveContainer" containerID="e4f756e9db5bff3f94813262ad0a2289f29e91302c724e04aa4342d6b4540bbd"
Sep 16 17:53:24.216916 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:53:24.216887    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4f756e9db5bff3f94813262ad0a2289f29e91302c724e04aa4342d6b4540bbd\": container with ID starting with e4f756e9db5bff3f94813262ad0a2289f29e91302c724e04aa4342d6b4540bbd not found: ID does not exist" containerID="e4f756e9db5bff3f94813262ad0a2289f29e91302c724e04aa4342d6b4540bbd"
Sep 16 17:53:24.217009 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:53:24.216930    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4f756e9db5bff3f94813262ad0a2289f29e91302c724e04aa4342d6b4540bbd"} err="failed to get container status \"e4f756e9db5bff3f94813262ad0a2289f29e91302c724e04aa4342d6b4540bbd\": rpc error: code = NotFound desc = could not find container \"e4f756e9db5bff3f94813262ad0a2289f29e91302c724e04aa4342d6b4540bbd\": container with ID starting with e4f756e9db5bff3f94813262ad0a2289f29e91302c724e04aa4342d6b4540bbd not found: ID does not exist"
Sep 16 17:53:24.217009 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:53:24.216956    2579 scope.go:117] "RemoveContainer" containerID="3367e50e47f261902be2e9bab286f77e4fcfe77eb31b6f844e865d348e960b11"
Sep 16 17:53:24.217009 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:53:24.216902    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-qcfjg"]
Sep 16 17:53:24.217233 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:53:24.217212    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3367e50e47f261902be2e9bab286f77e4fcfe77eb31b6f844e865d348e960b11\": container with ID starting with 3367e50e47f261902be2e9bab286f77e4fcfe77eb31b6f844e865d348e960b11 not found: ID does not exist" containerID="3367e50e47f261902be2e9bab286f77e4fcfe77eb31b6f844e865d348e960b11"
Sep 16 17:53:24.217300 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:53:24.217243    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3367e50e47f261902be2e9bab286f77e4fcfe77eb31b6f844e865d348e960b11"} err="failed to get container status \"3367e50e47f261902be2e9bab286f77e4fcfe77eb31b6f844e865d348e960b11\": rpc error: code = NotFound desc = could not find container \"3367e50e47f261902be2e9bab286f77e4fcfe77eb31b6f844e865d348e960b11\": container with ID starting with 3367e50e47f261902be2e9bab286f77e4fcfe77eb31b6f844e865d348e960b11 not found: ID does not exist"
Sep 16 17:53:24.217300 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:53:24.217266    2579 scope.go:117] "RemoveContainer" containerID="b0c10c8014ef1c32e61b27131dae367dac5c2984bed7505e251dc2c9704590e6"
Sep 16 17:53:24.217531 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:53:24.217513    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0c10c8014ef1c32e61b27131dae367dac5c2984bed7505e251dc2c9704590e6\": container with ID starting with b0c10c8014ef1c32e61b27131dae367dac5c2984bed7505e251dc2c9704590e6 not found: ID does not exist" containerID="b0c10c8014ef1c32e61b27131dae367dac5c2984bed7505e251dc2c9704590e6"
Sep 16 17:53:24.217571 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:53:24.217537    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0c10c8014ef1c32e61b27131dae367dac5c2984bed7505e251dc2c9704590e6"} err="failed to get container status \"b0c10c8014ef1c32e61b27131dae367dac5c2984bed7505e251dc2c9704590e6\": rpc error: code = NotFound desc = could not find container \"b0c10c8014ef1c32e61b27131dae367dac5c2984bed7505e251dc2c9704590e6\": container with ID starting with b0c10c8014ef1c32e61b27131dae367dac5c2984bed7505e251dc2c9704590e6 not found: ID does not exist"
Sep 16 17:53:26.099108 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:53:26.099073    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cdf1c8d9-1736-4ae0-ad71-ae966d09a6ce" path="/var/lib/kubelet/pods/cdf1c8d9-1736-4ae0-ad71-ae966d09a6ce/volumes"
Sep 16 17:53:32.750897 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:53:32.750859    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-pntj6"]
Sep 16 17:53:32.751305 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:53:32.751289    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cdf1c8d9-1736-4ae0-ad71-ae966d09a6ce" containerName="kube-rbac-proxy"
Sep 16 17:53:32.751354 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:53:32.751306    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="cdf1c8d9-1736-4ae0-ad71-ae966d09a6ce" containerName="kube-rbac-proxy"
Sep 16 17:53:32.751354 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:53:32.751319    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cdf1c8d9-1736-4ae0-ad71-ae966d09a6ce" containerName="storage-initializer"
Sep 16 17:53:32.751354 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:53:32.751324    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="cdf1c8d9-1736-4ae0-ad71-ae966d09a6ce" containerName="storage-initializer"
Sep 16 17:53:32.751354 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:53:32.751340    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2928c230-1840-489d-9a62-759097efe734" containerName="kserve-container"
Sep 16 17:53:32.751354 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:53:32.751348    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="2928c230-1840-489d-9a62-759097efe734" containerName="kserve-container"
Sep 16 17:53:32.751533 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:53:32.751361    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2928c230-1840-489d-9a62-759097efe734" containerName="kube-rbac-proxy"
Sep 16 17:53:32.751533 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:53:32.751368    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="2928c230-1840-489d-9a62-759097efe734" containerName="kube-rbac-proxy"
Sep 16 17:53:32.751533 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:53:32.751376    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cdf1c8d9-1736-4ae0-ad71-ae966d09a6ce" containerName="kserve-container"
Sep 16 17:53:32.751533 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:53:32.751382    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="cdf1c8d9-1736-4ae0-ad71-ae966d09a6ce" containerName="kserve-container"
Sep 16 17:53:32.751533 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:53:32.751409    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2928c230-1840-489d-9a62-759097efe734" containerName="storage-initializer"
Sep 16 17:53:32.751533 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:53:32.751414    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="2928c230-1840-489d-9a62-759097efe734" containerName="storage-initializer"
Sep 16 17:53:32.751533 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:53:32.751476    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="2928c230-1840-489d-9a62-759097efe734" containerName="kube-rbac-proxy"
Sep 16 17:53:32.751533 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:53:32.751486    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="cdf1c8d9-1736-4ae0-ad71-ae966d09a6ce" containerName="kube-rbac-proxy"
Sep 16 17:53:32.751533 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:53:32.751494    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="cdf1c8d9-1736-4ae0-ad71-ae966d09a6ce" containerName="kserve-container"
Sep 16 17:53:32.751533 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:53:32.751501    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="2928c230-1840-489d-9a62-759097efe734" containerName="kserve-container"
Sep 16 17:53:32.755120 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:53:32.755102    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-pntj6"
Sep 16 17:53:32.758757 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:53:32.758737    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 16 17:53:32.759702 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:53:32.759674    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-9bc9t\""
Sep 16 17:53:32.759702 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:53:32.759688    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 16 17:53:32.759702 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:53:32.759674    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-raw-cipn-predictor-serving-cert\""
Sep 16 17:53:32.759962 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:53:32.759771    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-raw-cipn-kube-rbac-proxy-sar-config\""
Sep 16 17:53:32.765281 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:53:32.765258    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-pntj6"]
Sep 16 17:53:32.944653 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:53:32.944618    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdnpz\" (UniqueName: \"kubernetes.io/projected/cec9bb14-5925-4d45-8b99-308423d7df49-kube-api-access-fdnpz\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-pntj6\" (UID: \"cec9bb14-5925-4d45-8b99-308423d7df49\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-pntj6"
Sep 16 17:53:32.944653 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:53:32.944659    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cec9bb14-5925-4d45-8b99-308423d7df49-proxy-tls\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-pntj6\" (UID: \"cec9bb14-5925-4d45-8b99-308423d7df49\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-pntj6"
Sep 16 17:53:32.944953 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:53:32.944766    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cec9bb14-5925-4d45-8b99-308423d7df49-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-pntj6\" (UID: \"cec9bb14-5925-4d45-8b99-308423d7df49\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-pntj6"
Sep 16 17:53:33.045716 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:53:33.045683    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cec9bb14-5925-4d45-8b99-308423d7df49-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-pntj6\" (UID: \"cec9bb14-5925-4d45-8b99-308423d7df49\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-pntj6"
Sep 16 17:53:33.045943 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:53:33.045746    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdnpz\" (UniqueName: \"kubernetes.io/projected/cec9bb14-5925-4d45-8b99-308423d7df49-kube-api-access-fdnpz\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-pntj6\" (UID: \"cec9bb14-5925-4d45-8b99-308423d7df49\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-pntj6"
Sep 16 17:53:33.045943 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:53:33.045780    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cec9bb14-5925-4d45-8b99-308423d7df49-proxy-tls\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-pntj6\" (UID: \"cec9bb14-5925-4d45-8b99-308423d7df49\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-pntj6"
Sep 16 17:53:33.046456 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:53:33.046430    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cec9bb14-5925-4d45-8b99-308423d7df49-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-pntj6\" (UID: \"cec9bb14-5925-4d45-8b99-308423d7df49\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-pntj6"
Sep 16 17:53:33.048241 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:53:33.048213    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cec9bb14-5925-4d45-8b99-308423d7df49-proxy-tls\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-pntj6\" (UID: \"cec9bb14-5925-4d45-8b99-308423d7df49\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-pntj6"
Sep 16 17:53:33.055476 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:53:33.055455    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdnpz\" (UniqueName: \"kubernetes.io/projected/cec9bb14-5925-4d45-8b99-308423d7df49-kube-api-access-fdnpz\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-pntj6\" (UID: \"cec9bb14-5925-4d45-8b99-308423d7df49\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-pntj6"
Sep 16 17:53:33.066374 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:53:33.066341    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-pntj6"
Sep 16 17:53:33.344353 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:53:33.344330    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-pntj6"]
Sep 16 17:53:33.346795 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:53:33.346771    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcec9bb14_5925_4d45_8b99_308423d7df49.slice/crio-7efd99cd1ff3484596816e525e0444eb24b697a0ccdddd9d4ed36b6ab2e84faa WatchSource:0}: Error finding container 7efd99cd1ff3484596816e525e0444eb24b697a0ccdddd9d4ed36b6ab2e84faa: Status 404 returned error can't find the container with id 7efd99cd1ff3484596816e525e0444eb24b697a0ccdddd9d4ed36b6ab2e84faa
Sep 16 17:53:34.238456 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:53:34.238421    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-pntj6" event={"ID":"cec9bb14-5925-4d45-8b99-308423d7df49","Type":"ContainerStarted","Data":"cdf7c26b848ccd549515fe1035640d55df9c54a274b61c9e716d95225874a20d"}
Sep 16 17:53:34.238869 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:53:34.238481    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-pntj6" event={"ID":"cec9bb14-5925-4d45-8b99-308423d7df49","Type":"ContainerStarted","Data":"43f53c8563e6f3e00e8c8e8d34b28e5b8121ed6ce1a57d78fa7f3f6774b7e2cc"}
Sep 16 17:53:34.238869 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:53:34.238501    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-pntj6" event={"ID":"cec9bb14-5925-4d45-8b99-308423d7df49","Type":"ContainerStarted","Data":"7efd99cd1ff3484596816e525e0444eb24b697a0ccdddd9d4ed36b6ab2e84faa"}
Sep 16 17:53:34.238869 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:53:34.238526    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-pntj6"
Sep 16 17:53:34.238869 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:53:34.238606    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-pntj6"
Sep 16 17:53:34.240491 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:53:34.240474    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-pntj6"
Sep 16 17:53:34.263564 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:53:34.263519    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-pntj6" podStartSLOduration=2.263504448 podStartE2EDuration="2.263504448s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:53:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:53:34.261796073 +0000 UTC m=+1726.731794208" watchObservedRunningTime="2026-09-16 17:53:34.263504448 +0000 UTC m=+1726.733502583"
Sep 16 17:53:41.253157 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:53:41.253128    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-pntj6"
Sep 16 17:53:42.760590 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:53:42.760560    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-577b6fff5f-9whgv"]
Sep 16 17:53:42.764499 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:53:42.764479    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-577b6fff5f-9whgv"
Sep 16 17:53:42.767442 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:53:42.767416    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 17:53:42.767573 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:53:42.767443    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-raw-cipn-kube-rbac-proxy-sar-config\""
Sep 16 17:53:42.767573 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:53:42.767522    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-raw-cipn-predictor-serving-cert\""
Sep 16 17:53:42.767687 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:53:42.767648    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 16 17:53:42.777936 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:53:42.777913    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-577b6fff5f-9whgv"]
Sep 16 17:53:42.823914 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:53:42.823884    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/90f138c9-d05c-4dc5-b562-d3602188a542-cabundle-cert\") pod \"isvc-logger-raw-cipn-predictor-577b6fff5f-9whgv\" (UID: \"90f138c9-d05c-4dc5-b562-d3602188a542\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-577b6fff5f-9whgv"
Sep 16 17:53:42.824074 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:53:42.823944    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/90f138c9-d05c-4dc5-b562-d3602188a542-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-cipn-predictor-577b6fff5f-9whgv\" (UID: \"90f138c9-d05c-4dc5-b562-d3602188a542\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-577b6fff5f-9whgv"
Sep 16 17:53:42.824074 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:53:42.823984    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvslx\" (UniqueName: \"kubernetes.io/projected/90f138c9-d05c-4dc5-b562-d3602188a542-kube-api-access-mvslx\") pod \"isvc-logger-raw-cipn-predictor-577b6fff5f-9whgv\" (UID: \"90f138c9-d05c-4dc5-b562-d3602188a542\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-577b6fff5f-9whgv"
Sep 16 17:53:42.824074 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:53:42.823998    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/90f138c9-d05c-4dc5-b562-d3602188a542-kserve-provision-location\") pod \"isvc-logger-raw-cipn-predictor-577b6fff5f-9whgv\" (UID: \"90f138c9-d05c-4dc5-b562-d3602188a542\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-577b6fff5f-9whgv"
Sep 16 17:53:42.824199 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:53:42.824073    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/90f138c9-d05c-4dc5-b562-d3602188a542-proxy-tls\") pod \"isvc-logger-raw-cipn-predictor-577b6fff5f-9whgv\" (UID: \"90f138c9-d05c-4dc5-b562-d3602188a542\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-577b6fff5f-9whgv"
Sep 16 17:53:42.924525 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:53:42.924491    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/90f138c9-d05c-4dc5-b562-d3602188a542-proxy-tls\") pod \"isvc-logger-raw-cipn-predictor-577b6fff5f-9whgv\" (UID: \"90f138c9-d05c-4dc5-b562-d3602188a542\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-577b6fff5f-9whgv"
Sep 16 17:53:42.924731 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:53:42.924539    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/90f138c9-d05c-4dc5-b562-d3602188a542-cabundle-cert\") pod \"isvc-logger-raw-cipn-predictor-577b6fff5f-9whgv\" (UID: \"90f138c9-d05c-4dc5-b562-d3602188a542\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-577b6fff5f-9whgv"
Sep 16 17:53:42.924731 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:53:42.924589    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/90f138c9-d05c-4dc5-b562-d3602188a542-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-cipn-predictor-577b6fff5f-9whgv\" (UID: \"90f138c9-d05c-4dc5-b562-d3602188a542\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-577b6fff5f-9whgv"
Sep 16 17:53:42.924731 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:53:42.924646    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvslx\" (UniqueName: \"kubernetes.io/projected/90f138c9-d05c-4dc5-b562-d3602188a542-kube-api-access-mvslx\") pod \"isvc-logger-raw-cipn-predictor-577b6fff5f-9whgv\" (UID: \"90f138c9-d05c-4dc5-b562-d3602188a542\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-577b6fff5f-9whgv"
Sep 16 17:53:42.924731 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:53:42.924662    2579 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-serving-cert: secret "isvc-logger-raw-cipn-predictor-serving-cert" not found
Sep 16 17:53:42.924731 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:53:42.924670    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/90f138c9-d05c-4dc5-b562-d3602188a542-kserve-provision-location\") pod \"isvc-logger-raw-cipn-predictor-577b6fff5f-9whgv\" (UID: \"90f138c9-d05c-4dc5-b562-d3602188a542\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-577b6fff5f-9whgv"
Sep 16 17:53:42.925042 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:53:42.924742    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/90f138c9-d05c-4dc5-b562-d3602188a542-proxy-tls podName:90f138c9-d05c-4dc5-b562-d3602188a542 nodeName:}" failed. No retries permitted until 2026-09-16 17:53:43.424720127 +0000 UTC m=+1735.894718317 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/90f138c9-d05c-4dc5-b562-d3602188a542-proxy-tls") pod "isvc-logger-raw-cipn-predictor-577b6fff5f-9whgv" (UID: "90f138c9-d05c-4dc5-b562-d3602188a542") : secret "isvc-logger-raw-cipn-predictor-serving-cert" not found
Sep 16 17:53:42.925213 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:53:42.925190    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/90f138c9-d05c-4dc5-b562-d3602188a542-kserve-provision-location\") pod \"isvc-logger-raw-cipn-predictor-577b6fff5f-9whgv\" (UID: \"90f138c9-d05c-4dc5-b562-d3602188a542\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-577b6fff5f-9whgv"
Sep 16 17:53:42.925521 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:53:42.925499    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/90f138c9-d05c-4dc5-b562-d3602188a542-cabundle-cert\") pod \"isvc-logger-raw-cipn-predictor-577b6fff5f-9whgv\" (UID: \"90f138c9-d05c-4dc5-b562-d3602188a542\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-577b6fff5f-9whgv"
Sep 16 17:53:42.925608 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:53:42.925584    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/90f138c9-d05c-4dc5-b562-d3602188a542-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-cipn-predictor-577b6fff5f-9whgv\" (UID: \"90f138c9-d05c-4dc5-b562-d3602188a542\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-577b6fff5f-9whgv"
Sep 16 17:53:42.933888 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:53:42.933868    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvslx\" (UniqueName: \"kubernetes.io/projected/90f138c9-d05c-4dc5-b562-d3602188a542-kube-api-access-mvslx\") pod \"isvc-logger-raw-cipn-predictor-577b6fff5f-9whgv\" (UID: \"90f138c9-d05c-4dc5-b562-d3602188a542\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-577b6fff5f-9whgv"
Sep 16 17:53:43.429211 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:53:43.429172    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/90f138c9-d05c-4dc5-b562-d3602188a542-proxy-tls\") pod \"isvc-logger-raw-cipn-predictor-577b6fff5f-9whgv\" (UID: \"90f138c9-d05c-4dc5-b562-d3602188a542\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-577b6fff5f-9whgv"
Sep 16 17:53:43.431725 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:53:43.431691    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/90f138c9-d05c-4dc5-b562-d3602188a542-proxy-tls\") pod \"isvc-logger-raw-cipn-predictor-577b6fff5f-9whgv\" (UID: \"90f138c9-d05c-4dc5-b562-d3602188a542\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-577b6fff5f-9whgv"
Sep 16 17:53:43.675715 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:53:43.675668    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-577b6fff5f-9whgv"
Sep 16 17:53:43.952265 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:53:43.952241    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-577b6fff5f-9whgv"]
Sep 16 17:53:43.954448 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:53:43.954411    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90f138c9_d05c_4dc5_b562_d3602188a542.slice/crio-b25c5e1122f99ddfa21137ce285eb1bd7cac3acdf0dd999d16e2345c682f3dde WatchSource:0}: Error finding container b25c5e1122f99ddfa21137ce285eb1bd7cac3acdf0dd999d16e2345c682f3dde: Status 404 returned error can't find the container with id b25c5e1122f99ddfa21137ce285eb1bd7cac3acdf0dd999d16e2345c682f3dde
Sep 16 17:53:44.298758 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:53:44.298716    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-577b6fff5f-9whgv" event={"ID":"90f138c9-d05c-4dc5-b562-d3602188a542","Type":"ContainerStarted","Data":"2d72af630da3349a406b2b456e52e479ffa156780f17b4034e388dc9095a2a32"}
Sep 16 17:53:44.298942 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:53:44.298832    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-577b6fff5f-9whgv" event={"ID":"90f138c9-d05c-4dc5-b562-d3602188a542","Type":"ContainerStarted","Data":"b25c5e1122f99ddfa21137ce285eb1bd7cac3acdf0dd999d16e2345c682f3dde"}
Sep 16 17:53:48.319762 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:53:48.319729    2579 generic.go:356] "Generic (PLEG): container finished" podID="90f138c9-d05c-4dc5-b562-d3602188a542" containerID="2d72af630da3349a406b2b456e52e479ffa156780f17b4034e388dc9095a2a32" exitCode=0
Sep 16 17:53:48.319762 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:53:48.319766    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-577b6fff5f-9whgv" event={"ID":"90f138c9-d05c-4dc5-b562-d3602188a542","Type":"ContainerDied","Data":"2d72af630da3349a406b2b456e52e479ffa156780f17b4034e388dc9095a2a32"}
Sep 16 17:53:49.327543 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:53:49.327507    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-577b6fff5f-9whgv" event={"ID":"90f138c9-d05c-4dc5-b562-d3602188a542","Type":"ContainerStarted","Data":"e539bb784862617de8718055e055c671132210519df44638c2178ebc928b178c"}
Sep 16 17:53:49.327936 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:53:49.327575    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-577b6fff5f-9whgv" event={"ID":"90f138c9-d05c-4dc5-b562-d3602188a542","Type":"ContainerStarted","Data":"e25b2f97cf00d428e00e69355a3e04916764fd7a470202328c7df83322fd3600"}
Sep 16 17:53:49.327936 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:53:49.327599    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-577b6fff5f-9whgv" event={"ID":"90f138c9-d05c-4dc5-b562-d3602188a542","Type":"ContainerStarted","Data":"7668c0d762cdf82d93ee4657ef7eb2c2083fa519b549bfbc9dfa1f4441b25791"}
Sep 16 17:53:49.327936 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:53:49.327924    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-577b6fff5f-9whgv"
Sep 16 17:53:49.328077 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:53:49.328056    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-577b6fff5f-9whgv"
Sep 16 17:53:49.329333 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:53:49.329311    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-577b6fff5f-9whgv" podUID="90f138c9-d05c-4dc5-b562-d3602188a542" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Sep 16 17:53:49.353018 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:53:49.352972    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-577b6fff5f-9whgv" podStartSLOduration=3.412388394 podStartE2EDuration="7.352958962s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.940570568s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:53:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:53:49.350969807 +0000 UTC m=+1741.820967954" watchObservedRunningTime="2026-09-16 17:53:49.352958962 +0000 UTC m=+1741.822957096"
Sep 16 17:53:50.332787 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:53:50.332754    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-577b6fff5f-9whgv"
Sep 16 17:53:50.333297 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:53:50.332877    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-577b6fff5f-9whgv" podUID="90f138c9-d05c-4dc5-b562-d3602188a542" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Sep 16 17:53:50.333881 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:53:50.333838    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-577b6fff5f-9whgv" podUID="90f138c9-d05c-4dc5-b562-d3602188a542" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 17:53:51.337335 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:53:51.337292    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-577b6fff5f-9whgv" podUID="90f138c9-d05c-4dc5-b562-d3602188a542" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Sep 16 17:53:51.337801 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:53:51.337778    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-577b6fff5f-9whgv" podUID="90f138c9-d05c-4dc5-b562-d3602188a542" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 17:53:56.342665 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:53:56.342637    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-577b6fff5f-9whgv"
Sep 16 17:53:56.343213 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:53:56.343173    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-577b6fff5f-9whgv" podUID="90f138c9-d05c-4dc5-b562-d3602188a542" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Sep 16 17:53:56.343619 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:53:56.343594    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-577b6fff5f-9whgv" podUID="90f138c9-d05c-4dc5-b562-d3602188a542" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 17:54:06.343724 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:54:06.343676    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-577b6fff5f-9whgv" podUID="90f138c9-d05c-4dc5-b562-d3602188a542" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Sep 16 17:54:06.344177 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:54:06.344125    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-577b6fff5f-9whgv" podUID="90f138c9-d05c-4dc5-b562-d3602188a542" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 17:54:16.343370 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:54:16.343324    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-577b6fff5f-9whgv" podUID="90f138c9-d05c-4dc5-b562-d3602188a542" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Sep 16 17:54:16.343756 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:54:16.343732    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-577b6fff5f-9whgv" podUID="90f138c9-d05c-4dc5-b562-d3602188a542" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 17:54:26.343424 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:54:26.343378    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-577b6fff5f-9whgv" podUID="90f138c9-d05c-4dc5-b562-d3602188a542" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Sep 16 17:54:26.343939 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:54:26.343818    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-577b6fff5f-9whgv" podUID="90f138c9-d05c-4dc5-b562-d3602188a542" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 17:54:36.343328 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:54:36.343282    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-577b6fff5f-9whgv" podUID="90f138c9-d05c-4dc5-b562-d3602188a542" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Sep 16 17:54:36.343762 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:54:36.343740    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-577b6fff5f-9whgv" podUID="90f138c9-d05c-4dc5-b562-d3602188a542" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 17:54:46.343568 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:54:46.343529    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-577b6fff5f-9whgv" podUID="90f138c9-d05c-4dc5-b562-d3602188a542" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Sep 16 17:54:46.344032 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:54:46.344001    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-577b6fff5f-9whgv" podUID="90f138c9-d05c-4dc5-b562-d3602188a542" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 17:54:56.343972 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:54:56.343925    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-577b6fff5f-9whgv" podUID="90f138c9-d05c-4dc5-b562-d3602188a542" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Sep 16 17:54:56.344377 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:54:56.344265    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-577b6fff5f-9whgv" podUID="90f138c9-d05c-4dc5-b562-d3602188a542" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 17:55:02.094084 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:55:02.094041    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-577b6fff5f-9whgv" podUID="90f138c9-d05c-4dc5-b562-d3602188a542" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Sep 16 17:55:02.094633 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:55:02.094415    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-577b6fff5f-9whgv" podUID="90f138c9-d05c-4dc5-b562-d3602188a542" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 17:55:12.099412 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:55:12.099384    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-577b6fff5f-9whgv"
Sep 16 17:55:12.099778 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:55:12.099474    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-577b6fff5f-9whgv"
Sep 16 17:55:17.847426 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:55:17.847393    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_message-dumper-raw-cipn-predictor-9774bf55d-pntj6_cec9bb14-5925-4d45-8b99-308423d7df49/kserve-container/0.log"
Sep 16 17:55:18.193814 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:55:18.193735    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-577b6fff5f-9whgv"]
Sep 16 17:55:18.194120 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:55:18.194093    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-577b6fff5f-9whgv" podUID="90f138c9-d05c-4dc5-b562-d3602188a542" containerName="kserve-container" containerID="cri-o://7668c0d762cdf82d93ee4657ef7eb2c2083fa519b549bfbc9dfa1f4441b25791" gracePeriod=30
Sep 16 17:55:18.194202 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:55:18.194177    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-577b6fff5f-9whgv" podUID="90f138c9-d05c-4dc5-b562-d3602188a542" containerName="agent" containerID="cri-o://e539bb784862617de8718055e055c671132210519df44638c2178ebc928b178c" gracePeriod=30
Sep 16 17:55:18.194265 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:55:18.194195    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-577b6fff5f-9whgv" podUID="90f138c9-d05c-4dc5-b562-d3602188a542" containerName="kube-rbac-proxy" containerID="cri-o://e25b2f97cf00d428e00e69355a3e04916764fd7a470202328c7df83322fd3600" gracePeriod=30
Sep 16 17:55:18.257486 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:55:18.257446    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-pntj6"]
Sep 16 17:55:18.257739 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:55:18.257711    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-pntj6" podUID="cec9bb14-5925-4d45-8b99-308423d7df49" containerName="kserve-container" containerID="cri-o://43f53c8563e6f3e00e8c8e8d34b28e5b8121ed6ce1a57d78fa7f3f6774b7e2cc" gracePeriod=30
Sep 16 17:55:18.257832 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:55:18.257750    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-pntj6" podUID="cec9bb14-5925-4d45-8b99-308423d7df49" containerName="kube-rbac-proxy" containerID="cri-o://cdf7c26b848ccd549515fe1035640d55df9c54a274b61c9e716d95225874a20d" gracePeriod=30
Sep 16 17:55:18.518352 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:55:18.518326    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-pntj6"
Sep 16 17:55:18.635795 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:55:18.635761    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fdnpz\" (UniqueName: \"kubernetes.io/projected/cec9bb14-5925-4d45-8b99-308423d7df49-kube-api-access-fdnpz\") pod \"cec9bb14-5925-4d45-8b99-308423d7df49\" (UID: \"cec9bb14-5925-4d45-8b99-308423d7df49\") "
Sep 16 17:55:18.635795 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:55:18.635795    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cec9bb14-5925-4d45-8b99-308423d7df49-kube-rbac-proxy-sar-config\") pod \"cec9bb14-5925-4d45-8b99-308423d7df49\" (UID: \"cec9bb14-5925-4d45-8b99-308423d7df49\") "
Sep 16 17:55:18.636066 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:55:18.635955    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cec9bb14-5925-4d45-8b99-308423d7df49-proxy-tls\") pod \"cec9bb14-5925-4d45-8b99-308423d7df49\" (UID: \"cec9bb14-5925-4d45-8b99-308423d7df49\") "
Sep 16 17:55:18.636307 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:55:18.636283    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cec9bb14-5925-4d45-8b99-308423d7df49-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "cec9bb14-5925-4d45-8b99-308423d7df49" (UID: "cec9bb14-5925-4d45-8b99-308423d7df49"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 17:55:18.638052 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:55:18.638023    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cec9bb14-5925-4d45-8b99-308423d7df49-kube-api-access-fdnpz" (OuterVolumeSpecName: "kube-api-access-fdnpz") pod "cec9bb14-5925-4d45-8b99-308423d7df49" (UID: "cec9bb14-5925-4d45-8b99-308423d7df49"). InnerVolumeSpecName "kube-api-access-fdnpz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 17:55:18.638151 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:55:18.638103    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cec9bb14-5925-4d45-8b99-308423d7df49-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "cec9bb14-5925-4d45-8b99-308423d7df49" (UID: "cec9bb14-5925-4d45-8b99-308423d7df49"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 17:55:18.737311 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:55:18.737260    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cec9bb14-5925-4d45-8b99-308423d7df49-proxy-tls\") on node \"ip-10-0-138-73.ec2.internal\" DevicePath \"\""
Sep 16 17:55:18.737311 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:55:18.737304    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fdnpz\" (UniqueName: \"kubernetes.io/projected/cec9bb14-5925-4d45-8b99-308423d7df49-kube-api-access-fdnpz\") on node \"ip-10-0-138-73.ec2.internal\" DevicePath \"\""
Sep 16 17:55:18.737311 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:55:18.737317    2579 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cec9bb14-5925-4d45-8b99-308423d7df49-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-73.ec2.internal\" DevicePath \"\""
Sep 16 17:55:18.764451 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:55:18.764363    2579 generic.go:356] "Generic (PLEG): container finished" podID="90f138c9-d05c-4dc5-b562-d3602188a542" containerID="e25b2f97cf00d428e00e69355a3e04916764fd7a470202328c7df83322fd3600" exitCode=2
Sep 16 17:55:18.764451 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:55:18.764436    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-577b6fff5f-9whgv" event={"ID":"90f138c9-d05c-4dc5-b562-d3602188a542","Type":"ContainerDied","Data":"e25b2f97cf00d428e00e69355a3e04916764fd7a470202328c7df83322fd3600"}
Sep 16 17:55:18.766101 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:55:18.766079    2579 generic.go:356] "Generic (PLEG): container finished" podID="cec9bb14-5925-4d45-8b99-308423d7df49" containerID="cdf7c26b848ccd549515fe1035640d55df9c54a274b61c9e716d95225874a20d" exitCode=2
Sep 16 17:55:18.766101 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:55:18.766099    2579 generic.go:356] "Generic (PLEG): container finished" podID="cec9bb14-5925-4d45-8b99-308423d7df49" containerID="43f53c8563e6f3e00e8c8e8d34b28e5b8121ed6ce1a57d78fa7f3f6774b7e2cc" exitCode=2
Sep 16 17:55:18.766252 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:55:18.766127    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-pntj6"
Sep 16 17:55:18.766252 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:55:18.766163    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-pntj6" event={"ID":"cec9bb14-5925-4d45-8b99-308423d7df49","Type":"ContainerDied","Data":"cdf7c26b848ccd549515fe1035640d55df9c54a274b61c9e716d95225874a20d"}
Sep 16 17:55:18.766252 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:55:18.766224    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-pntj6" event={"ID":"cec9bb14-5925-4d45-8b99-308423d7df49","Type":"ContainerDied","Data":"43f53c8563e6f3e00e8c8e8d34b28e5b8121ed6ce1a57d78fa7f3f6774b7e2cc"}
Sep 16 17:55:18.766354 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:55:18.766254    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-pntj6" event={"ID":"cec9bb14-5925-4d45-8b99-308423d7df49","Type":"ContainerDied","Data":"7efd99cd1ff3484596816e525e0444eb24b697a0ccdddd9d4ed36b6ab2e84faa"}
Sep 16 17:55:18.766354 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:55:18.766284    2579 scope.go:117] "RemoveContainer" containerID="cdf7c26b848ccd549515fe1035640d55df9c54a274b61c9e716d95225874a20d"
Sep 16 17:55:18.775590 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:55:18.775569    2579 scope.go:117] "RemoveContainer" containerID="43f53c8563e6f3e00e8c8e8d34b28e5b8121ed6ce1a57d78fa7f3f6774b7e2cc"
Sep 16 17:55:18.783095 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:55:18.783079    2579 scope.go:117] "RemoveContainer" containerID="cdf7c26b848ccd549515fe1035640d55df9c54a274b61c9e716d95225874a20d"
Sep 16 17:55:18.783359 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:55:18.783339    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cdf7c26b848ccd549515fe1035640d55df9c54a274b61c9e716d95225874a20d\": container with ID starting with cdf7c26b848ccd549515fe1035640d55df9c54a274b61c9e716d95225874a20d not found: ID does not exist" containerID="cdf7c26b848ccd549515fe1035640d55df9c54a274b61c9e716d95225874a20d"
Sep 16 17:55:18.783448 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:55:18.783371    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cdf7c26b848ccd549515fe1035640d55df9c54a274b61c9e716d95225874a20d"} err="failed to get container status \"cdf7c26b848ccd549515fe1035640d55df9c54a274b61c9e716d95225874a20d\": rpc error: code = NotFound desc = could not find container \"cdf7c26b848ccd549515fe1035640d55df9c54a274b61c9e716d95225874a20d\": container with ID starting with cdf7c26b848ccd549515fe1035640d55df9c54a274b61c9e716d95225874a20d not found: ID does not exist"
Sep 16 17:55:18.783448 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:55:18.783396    2579 scope.go:117] "RemoveContainer" containerID="43f53c8563e6f3e00e8c8e8d34b28e5b8121ed6ce1a57d78fa7f3f6774b7e2cc"
Sep 16 17:55:18.783644 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:55:18.783626    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43f53c8563e6f3e00e8c8e8d34b28e5b8121ed6ce1a57d78fa7f3f6774b7e2cc\": container with ID starting with 43f53c8563e6f3e00e8c8e8d34b28e5b8121ed6ce1a57d78fa7f3f6774b7e2cc not found: ID does not exist" containerID="43f53c8563e6f3e00e8c8e8d34b28e5b8121ed6ce1a57d78fa7f3f6774b7e2cc"
Sep 16 17:55:18.783684 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:55:18.783649    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43f53c8563e6f3e00e8c8e8d34b28e5b8121ed6ce1a57d78fa7f3f6774b7e2cc"} err="failed to get container status \"43f53c8563e6f3e00e8c8e8d34b28e5b8121ed6ce1a57d78fa7f3f6774b7e2cc\": rpc error: code = NotFound desc = could not find container \"43f53c8563e6f3e00e8c8e8d34b28e5b8121ed6ce1a57d78fa7f3f6774b7e2cc\": container with ID starting with 43f53c8563e6f3e00e8c8e8d34b28e5b8121ed6ce1a57d78fa7f3f6774b7e2cc not found: ID does not exist"
Sep 16 17:55:18.783684 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:55:18.783664    2579 scope.go:117] "RemoveContainer" containerID="cdf7c26b848ccd549515fe1035640d55df9c54a274b61c9e716d95225874a20d"
Sep 16 17:55:18.783886 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:55:18.783868    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cdf7c26b848ccd549515fe1035640d55df9c54a274b61c9e716d95225874a20d"} err="failed to get container status \"cdf7c26b848ccd549515fe1035640d55df9c54a274b61c9e716d95225874a20d\": rpc error: code = NotFound desc = could not find container \"cdf7c26b848ccd549515fe1035640d55df9c54a274b61c9e716d95225874a20d\": container with ID starting with cdf7c26b848ccd549515fe1035640d55df9c54a274b61c9e716d95225874a20d not found: ID does not exist"
Sep 16 17:55:18.783943 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:55:18.783888    2579 scope.go:117] "RemoveContainer" containerID="43f53c8563e6f3e00e8c8e8d34b28e5b8121ed6ce1a57d78fa7f3f6774b7e2cc"
Sep 16 17:55:18.784086 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:55:18.784068    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43f53c8563e6f3e00e8c8e8d34b28e5b8121ed6ce1a57d78fa7f3f6774b7e2cc"} err="failed to get container status \"43f53c8563e6f3e00e8c8e8d34b28e5b8121ed6ce1a57d78fa7f3f6774b7e2cc\": rpc error: code = NotFound desc = could not find container \"43f53c8563e6f3e00e8c8e8d34b28e5b8121ed6ce1a57d78fa7f3f6774b7e2cc\": container with ID starting with 43f53c8563e6f3e00e8c8e8d34b28e5b8121ed6ce1a57d78fa7f3f6774b7e2cc not found: ID does not exist"
Sep 16 17:55:18.793196 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:55:18.793151    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-pntj6"]
Sep 16 17:55:18.794917 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:55:18.794897    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-pntj6"]
Sep 16 17:55:20.098746 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:55:20.098715    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cec9bb14-5925-4d45-8b99-308423d7df49" path="/var/lib/kubelet/pods/cec9bb14-5925-4d45-8b99-308423d7df49/volumes"
Sep 16 17:55:21.338153 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:55:21.338117    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-577b6fff5f-9whgv" podUID="90f138c9-d05c-4dc5-b562-d3602188a542" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.49:8643/healthz\": dial tcp 10.134.0.49:8643: connect: connection refused"
Sep 16 17:55:22.094089 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:55:22.094038    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-577b6fff5f-9whgv" podUID="90f138c9-d05c-4dc5-b562-d3602188a542" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Sep 16 17:55:22.094415 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:55:22.094385    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-577b6fff5f-9whgv" podUID="90f138c9-d05c-4dc5-b562-d3602188a542" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 17:55:23.792956 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:55:23.792925    2579 generic.go:356] "Generic (PLEG): container finished" podID="90f138c9-d05c-4dc5-b562-d3602188a542" containerID="7668c0d762cdf82d93ee4657ef7eb2c2083fa519b549bfbc9dfa1f4441b25791" exitCode=0
Sep 16 17:55:23.793328 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:55:23.792967    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-577b6fff5f-9whgv" event={"ID":"90f138c9-d05c-4dc5-b562-d3602188a542","Type":"ContainerDied","Data":"7668c0d762cdf82d93ee4657ef7eb2c2083fa519b549bfbc9dfa1f4441b25791"}
Sep 16 17:55:26.338052 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:55:26.338010    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-577b6fff5f-9whgv" podUID="90f138c9-d05c-4dc5-b562-d3602188a542" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.49:8643/healthz\": dial tcp 10.134.0.49:8643: connect: connection refused"
Sep 16 17:55:31.338121 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:55:31.338078    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-577b6fff5f-9whgv" podUID="90f138c9-d05c-4dc5-b562-d3602188a542" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.49:8643/healthz\": dial tcp 10.134.0.49:8643: connect: connection refused"
Sep 16 17:55:31.338570 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:55:31.338198    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-577b6fff5f-9whgv"
Sep 16 17:55:32.094832 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:55:32.094780    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-577b6fff5f-9whgv" podUID="90f138c9-d05c-4dc5-b562-d3602188a542" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Sep 16 17:55:32.095136 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:55:32.095110    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-577b6fff5f-9whgv" podUID="90f138c9-d05c-4dc5-b562-d3602188a542" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 17:55:34.572379 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:55:34.572306    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-577b6fff5f-9whgv_90f138c9-d05c-4dc5-b562-d3602188a542/kserve-container/0.log"
Sep 16 17:55:34.597281 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:55:34.597255    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-577b6fff5f-9whgv_90f138c9-d05c-4dc5-b562-d3602188a542/kube-rbac-proxy/0.log"
Sep 16 17:55:34.608446 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:55:34.608423    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-577b6fff5f-9whgv_90f138c9-d05c-4dc5-b562-d3602188a542/agent/0.log"
Sep 16 17:55:34.619058 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:55:34.619040    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-577b6fff5f-9whgv_90f138c9-d05c-4dc5-b562-d3602188a542/storage-initializer/0.log"
Sep 16 17:55:35.431138 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:55:35.431099    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-577b6fff5f-9whgv_90f138c9-d05c-4dc5-b562-d3602188a542/kserve-container/0.log"
Sep 16 17:55:35.443945 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:55:35.443919    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-577b6fff5f-9whgv_90f138c9-d05c-4dc5-b562-d3602188a542/kube-rbac-proxy/0.log"
Sep 16 17:55:35.456412 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:55:35.456372    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-577b6fff5f-9whgv_90f138c9-d05c-4dc5-b562-d3602188a542/agent/0.log"
Sep 16 17:55:35.469768 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:55:35.469733    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-577b6fff5f-9whgv_90f138c9-d05c-4dc5-b562-d3602188a542/storage-initializer/0.log"
Sep 16 17:55:36.331491 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:55:36.331445    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-577b6fff5f-9whgv_90f138c9-d05c-4dc5-b562-d3602188a542/kserve-container/0.log"
Sep 16 17:55:36.337604 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:55:36.337573    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-577b6fff5f-9whgv" podUID="90f138c9-d05c-4dc5-b562-d3602188a542" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.49:8643/healthz\": dial tcp 10.134.0.49:8643: connect: connection refused"
Sep 16 17:55:36.355520 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:55:36.355493    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-577b6fff5f-9whgv_90f138c9-d05c-4dc5-b562-d3602188a542/kube-rbac-proxy/0.log"
Sep 16 17:55:36.380400 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:55:36.380373    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-577b6fff5f-9whgv_90f138c9-d05c-4dc5-b562-d3602188a542/agent/0.log"
Sep 16 17:55:36.453686 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:55:36.453657    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-577b6fff5f-9whgv_90f138c9-d05c-4dc5-b562-d3602188a542/storage-initializer/0.log"
Sep 16 17:55:37.294469 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:55:37.294439    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-577b6fff5f-9whgv_90f138c9-d05c-4dc5-b562-d3602188a542/kserve-container/0.log"
Sep 16 17:55:37.310807 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:55:37.310778    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-577b6fff5f-9whgv_90f138c9-d05c-4dc5-b562-d3602188a542/kube-rbac-proxy/0.log"
Sep 16 17:55:37.324623 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:55:37.324583    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-577b6fff5f-9whgv_90f138c9-d05c-4dc5-b562-d3602188a542/agent/0.log"
Sep 16 17:55:37.336731 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:55:37.336707    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-577b6fff5f-9whgv_90f138c9-d05c-4dc5-b562-d3602188a542/storage-initializer/0.log"
Sep 16 17:55:38.226023 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:55:38.225995    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-577b6fff5f-9whgv_90f138c9-d05c-4dc5-b562-d3602188a542/kserve-container/0.log"
Sep 16 17:55:38.239148 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:55:38.239124    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-577b6fff5f-9whgv_90f138c9-d05c-4dc5-b562-d3602188a542/kube-rbac-proxy/0.log"
Sep 16 17:55:38.249994 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:55:38.249966    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-577b6fff5f-9whgv_90f138c9-d05c-4dc5-b562-d3602188a542/agent/0.log"
Sep 16 17:55:38.260862 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:55:38.260819    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-577b6fff5f-9whgv_90f138c9-d05c-4dc5-b562-d3602188a542/storage-initializer/0.log"
Sep 16 17:55:39.064898 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:55:39.064860    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-577b6fff5f-9whgv_90f138c9-d05c-4dc5-b562-d3602188a542/kserve-container/0.log"
Sep 16 17:55:39.079395 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:55:39.079372    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-577b6fff5f-9whgv_90f138c9-d05c-4dc5-b562-d3602188a542/kube-rbac-proxy/0.log"
Sep 16 17:55:39.091302 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:55:39.091255    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-577b6fff5f-9whgv_90f138c9-d05c-4dc5-b562-d3602188a542/agent/0.log"
Sep 16 17:55:39.109690 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:55:39.109666    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-577b6fff5f-9whgv_90f138c9-d05c-4dc5-b562-d3602188a542/storage-initializer/0.log"
Sep 16 17:55:39.985177 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:55:39.985132    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-577b6fff5f-9whgv_90f138c9-d05c-4dc5-b562-d3602188a542/kserve-container/0.log"
Sep 16 17:55:39.997320 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:55:39.997299    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-577b6fff5f-9whgv_90f138c9-d05c-4dc5-b562-d3602188a542/kube-rbac-proxy/0.log"
Sep 16 17:55:40.010069 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:55:40.010048    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-577b6fff5f-9whgv_90f138c9-d05c-4dc5-b562-d3602188a542/agent/0.log"
Sep 16 17:55:40.023288 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:55:40.023251    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-577b6fff5f-9whgv_90f138c9-d05c-4dc5-b562-d3602188a542/storage-initializer/0.log"
Sep 16 17:55:40.872101 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:55:40.872067    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-577b6fff5f-9whgv_90f138c9-d05c-4dc5-b562-d3602188a542/kserve-container/0.log"
Sep 16 17:55:40.886321 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:55:40.886292    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-577b6fff5f-9whgv_90f138c9-d05c-4dc5-b562-d3602188a542/kube-rbac-proxy/0.log"
Sep 16 17:55:40.907186 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:55:40.907154    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-577b6fff5f-9whgv_90f138c9-d05c-4dc5-b562-d3602188a542/agent/0.log"
Sep 16 17:55:40.922042 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:55:40.922013    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-577b6fff5f-9whgv_90f138c9-d05c-4dc5-b562-d3602188a542/storage-initializer/0.log"
Sep 16 17:55:41.337808 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:55:41.337770    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-577b6fff5f-9whgv" podUID="90f138c9-d05c-4dc5-b562-d3602188a542" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.49:8643/healthz\": dial tcp 10.134.0.49:8643: connect: connection refused"
Sep 16 17:55:41.756279 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:55:41.756196    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-577b6fff5f-9whgv_90f138c9-d05c-4dc5-b562-d3602188a542/kserve-container/0.log"
Sep 16 17:55:41.774409 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:55:41.774373    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-577b6fff5f-9whgv_90f138c9-d05c-4dc5-b562-d3602188a542/kube-rbac-proxy/0.log"
Sep 16 17:55:41.786436 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:55:41.786393    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-577b6fff5f-9whgv_90f138c9-d05c-4dc5-b562-d3602188a542/agent/0.log"
Sep 16 17:55:41.801245 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:55:41.801223    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-577b6fff5f-9whgv_90f138c9-d05c-4dc5-b562-d3602188a542/storage-initializer/0.log"
Sep 16 17:55:42.094301 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:55:42.094254    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-577b6fff5f-9whgv" podUID="90f138c9-d05c-4dc5-b562-d3602188a542" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Sep 16 17:55:42.094699 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:55:42.094501    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-577b6fff5f-9whgv" podUID="90f138c9-d05c-4dc5-b562-d3602188a542" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 17:55:42.099918 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:55:42.099891    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-577b6fff5f-9whgv"
Sep 16 17:55:42.100077 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:55:42.100003    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-577b6fff5f-9whgv"
Sep 16 17:55:42.615330 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:55:42.615303    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-577b6fff5f-9whgv_90f138c9-d05c-4dc5-b562-d3602188a542/kserve-container/0.log"
Sep 16 17:55:42.629273 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:55:42.629237    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-577b6fff5f-9whgv_90f138c9-d05c-4dc5-b562-d3602188a542/kube-rbac-proxy/0.log"
Sep 16 17:55:42.644747 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:55:42.644717    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-577b6fff5f-9whgv_90f138c9-d05c-4dc5-b562-d3602188a542/agent/0.log"
Sep 16 17:55:42.662523 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:55:42.662499    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-577b6fff5f-9whgv_90f138c9-d05c-4dc5-b562-d3602188a542/storage-initializer/0.log"
Sep 16 17:55:43.505053 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:55:43.505021    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-577b6fff5f-9whgv_90f138c9-d05c-4dc5-b562-d3602188a542/kserve-container/0.log"
Sep 16 17:55:43.519613 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:55:43.519584    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-577b6fff5f-9whgv_90f138c9-d05c-4dc5-b562-d3602188a542/kube-rbac-proxy/0.log"
Sep 16 17:55:43.533840 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:55:43.533816    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-577b6fff5f-9whgv_90f138c9-d05c-4dc5-b562-d3602188a542/agent/0.log"
Sep 16 17:55:43.548470 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:55:43.548447    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-577b6fff5f-9whgv_90f138c9-d05c-4dc5-b562-d3602188a542/storage-initializer/0.log"
Sep 16 17:55:44.362396 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:55:44.362371    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-577b6fff5f-9whgv_90f138c9-d05c-4dc5-b562-d3602188a542/kserve-container/0.log"
Sep 16 17:55:44.376486 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:55:44.376458    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-577b6fff5f-9whgv_90f138c9-d05c-4dc5-b562-d3602188a542/kube-rbac-proxy/0.log"
Sep 16 17:55:44.398493 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:55:44.398466    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-577b6fff5f-9whgv_90f138c9-d05c-4dc5-b562-d3602188a542/agent/0.log"
Sep 16 17:55:44.414796 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:55:44.414766    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-577b6fff5f-9whgv_90f138c9-d05c-4dc5-b562-d3602188a542/storage-initializer/0.log"
Sep 16 17:55:45.374332 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:55:45.374306    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-577b6fff5f-9whgv_90f138c9-d05c-4dc5-b562-d3602188a542/kserve-container/0.log"
Sep 16 17:55:45.388770 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:55:45.388738    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-577b6fff5f-9whgv_90f138c9-d05c-4dc5-b562-d3602188a542/kube-rbac-proxy/0.log"
Sep 16 17:55:45.402728 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:55:45.402698    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-577b6fff5f-9whgv_90f138c9-d05c-4dc5-b562-d3602188a542/agent/0.log"
Sep 16 17:55:45.415664 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:55:45.415641    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-577b6fff5f-9whgv_90f138c9-d05c-4dc5-b562-d3602188a542/storage-initializer/0.log"
Sep 16 17:55:46.337829 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:55:46.337784    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-577b6fff5f-9whgv" podUID="90f138c9-d05c-4dc5-b562-d3602188a542" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.49:8643/healthz\": dial tcp 10.134.0.49:8643: connect: connection refused"
Sep 16 17:55:46.379930 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:55:46.379901    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-577b6fff5f-9whgv_90f138c9-d05c-4dc5-b562-d3602188a542/kserve-container/0.log"
Sep 16 17:55:46.402197 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:55:46.402167    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-577b6fff5f-9whgv_90f138c9-d05c-4dc5-b562-d3602188a542/kube-rbac-proxy/0.log"
Sep 16 17:55:46.425222 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:55:46.425181    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-577b6fff5f-9whgv_90f138c9-d05c-4dc5-b562-d3602188a542/agent/0.log"
Sep 16 17:55:46.444269 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:55:46.444241    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-577b6fff5f-9whgv_90f138c9-d05c-4dc5-b562-d3602188a542/storage-initializer/0.log"
Sep 16 17:55:48.392572 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:55:48.392547    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-577b6fff5f-9whgv"
Sep 16 17:55:48.496714 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:55:48.496622    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/90f138c9-d05c-4dc5-b562-d3602188a542-kube-rbac-proxy-sar-config\") pod \"90f138c9-d05c-4dc5-b562-d3602188a542\" (UID: \"90f138c9-d05c-4dc5-b562-d3602188a542\") "
Sep 16 17:55:48.496714 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:55:48.496668    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/90f138c9-d05c-4dc5-b562-d3602188a542-cabundle-cert\") pod \"90f138c9-d05c-4dc5-b562-d3602188a542\" (UID: \"90f138c9-d05c-4dc5-b562-d3602188a542\") "
Sep 16 17:55:48.496714 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:55:48.496707    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/90f138c9-d05c-4dc5-b562-d3602188a542-kserve-provision-location\") pod \"90f138c9-d05c-4dc5-b562-d3602188a542\" (UID: \"90f138c9-d05c-4dc5-b562-d3602188a542\") "
Sep 16 17:55:48.497040 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:55:48.496732    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/90f138c9-d05c-4dc5-b562-d3602188a542-proxy-tls\") pod \"90f138c9-d05c-4dc5-b562-d3602188a542\" (UID: \"90f138c9-d05c-4dc5-b562-d3602188a542\") "
Sep 16 17:55:48.497040 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:55:48.496790    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mvslx\" (UniqueName: \"kubernetes.io/projected/90f138c9-d05c-4dc5-b562-d3602188a542-kube-api-access-mvslx\") pod \"90f138c9-d05c-4dc5-b562-d3602188a542\" (UID: \"90f138c9-d05c-4dc5-b562-d3602188a542\") "
Sep 16 17:55:48.497148 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:55:48.497080    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90f138c9-d05c-4dc5-b562-d3602188a542-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "90f138c9-d05c-4dc5-b562-d3602188a542" (UID: "90f138c9-d05c-4dc5-b562-d3602188a542"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 17:55:48.497148 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:55:48.497116    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90f138c9-d05c-4dc5-b562-d3602188a542-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "90f138c9-d05c-4dc5-b562-d3602188a542" (UID: "90f138c9-d05c-4dc5-b562-d3602188a542"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:55:48.497228 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:55:48.497143    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90f138c9-d05c-4dc5-b562-d3602188a542-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "90f138c9-d05c-4dc5-b562-d3602188a542" (UID: "90f138c9-d05c-4dc5-b562-d3602188a542"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 17:55:48.499014 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:55:48.498983    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90f138c9-d05c-4dc5-b562-d3602188a542-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "90f138c9-d05c-4dc5-b562-d3602188a542" (UID: "90f138c9-d05c-4dc5-b562-d3602188a542"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 17:55:48.499122 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:55:48.499045    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90f138c9-d05c-4dc5-b562-d3602188a542-kube-api-access-mvslx" (OuterVolumeSpecName: "kube-api-access-mvslx") pod "90f138c9-d05c-4dc5-b562-d3602188a542" (UID: "90f138c9-d05c-4dc5-b562-d3602188a542"). InnerVolumeSpecName "kube-api-access-mvslx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 17:55:48.598054 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:55:48.598019    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mvslx\" (UniqueName: \"kubernetes.io/projected/90f138c9-d05c-4dc5-b562-d3602188a542-kube-api-access-mvslx\") on node \"ip-10-0-138-73.ec2.internal\" DevicePath \"\""
Sep 16 17:55:48.598054 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:55:48.598047    2579 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/90f138c9-d05c-4dc5-b562-d3602188a542-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-73.ec2.internal\" DevicePath \"\""
Sep 16 17:55:48.598054 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:55:48.598059    2579 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/90f138c9-d05c-4dc5-b562-d3602188a542-cabundle-cert\") on node \"ip-10-0-138-73.ec2.internal\" DevicePath \"\""
Sep 16 17:55:48.598291 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:55:48.598068    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/90f138c9-d05c-4dc5-b562-d3602188a542-kserve-provision-location\") on node \"ip-10-0-138-73.ec2.internal\" DevicePath \"\""
Sep 16 17:55:48.598291 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:55:48.598078    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/90f138c9-d05c-4dc5-b562-d3602188a542-proxy-tls\") on node \"ip-10-0-138-73.ec2.internal\" DevicePath \"\""
Sep 16 17:55:48.920517 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:55:48.920482    2579 generic.go:356] "Generic (PLEG): container finished" podID="90f138c9-d05c-4dc5-b562-d3602188a542" containerID="e539bb784862617de8718055e055c671132210519df44638c2178ebc928b178c" exitCode=137
Sep 16 17:55:48.920708 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:55:48.920543    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-577b6fff5f-9whgv"
Sep 16 17:55:48.920708 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:55:48.920574    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-577b6fff5f-9whgv" event={"ID":"90f138c9-d05c-4dc5-b562-d3602188a542","Type":"ContainerDied","Data":"e539bb784862617de8718055e055c671132210519df44638c2178ebc928b178c"}
Sep 16 17:55:48.920708 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:55:48.920642    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-577b6fff5f-9whgv" event={"ID":"90f138c9-d05c-4dc5-b562-d3602188a542","Type":"ContainerDied","Data":"b25c5e1122f99ddfa21137ce285eb1bd7cac3acdf0dd999d16e2345c682f3dde"}
Sep 16 17:55:48.920708 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:55:48.920675    2579 scope.go:117] "RemoveContainer" containerID="e539bb784862617de8718055e055c671132210519df44638c2178ebc928b178c"
Sep 16 17:55:48.929974 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:55:48.929954    2579 scope.go:117] "RemoveContainer" containerID="e25b2f97cf00d428e00e69355a3e04916764fd7a470202328c7df83322fd3600"
Sep 16 17:55:48.937773 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:55:48.937751    2579 scope.go:117] "RemoveContainer" containerID="7668c0d762cdf82d93ee4657ef7eb2c2083fa519b549bfbc9dfa1f4441b25791"
Sep 16 17:55:48.945225 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:55:48.945208    2579 scope.go:117] "RemoveContainer" containerID="2d72af630da3349a406b2b456e52e479ffa156780f17b4034e388dc9095a2a32"
Sep 16 17:55:48.953486 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:55:48.953466    2579 scope.go:117] "RemoveContainer" containerID="e539bb784862617de8718055e055c671132210519df44638c2178ebc928b178c"
Sep 16 17:55:48.953781 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:55:48.953757    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e539bb784862617de8718055e055c671132210519df44638c2178ebc928b178c\": container with ID starting with e539bb784862617de8718055e055c671132210519df44638c2178ebc928b178c not found: ID does not exist" containerID="e539bb784862617de8718055e055c671132210519df44638c2178ebc928b178c"
Sep 16 17:55:48.953885 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:55:48.953789    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e539bb784862617de8718055e055c671132210519df44638c2178ebc928b178c"} err="failed to get container status \"e539bb784862617de8718055e055c671132210519df44638c2178ebc928b178c\": rpc error: code = NotFound desc = could not find container \"e539bb784862617de8718055e055c671132210519df44638c2178ebc928b178c\": container with ID starting with e539bb784862617de8718055e055c671132210519df44638c2178ebc928b178c not found: ID does not exist"
Sep 16 17:55:48.953885 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:55:48.953807    2579 scope.go:117] "RemoveContainer" containerID="e25b2f97cf00d428e00e69355a3e04916764fd7a470202328c7df83322fd3600"
Sep 16 17:55:48.954131 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:55:48.954114    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e25b2f97cf00d428e00e69355a3e04916764fd7a470202328c7df83322fd3600\": container with ID starting with e25b2f97cf00d428e00e69355a3e04916764fd7a470202328c7df83322fd3600 not found: ID does not exist" containerID="e25b2f97cf00d428e00e69355a3e04916764fd7a470202328c7df83322fd3600"
Sep 16 17:55:48.954205 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:55:48.954133    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e25b2f97cf00d428e00e69355a3e04916764fd7a470202328c7df83322fd3600"} err="failed to get container status \"e25b2f97cf00d428e00e69355a3e04916764fd7a470202328c7df83322fd3600\": rpc error: code = NotFound desc = could not find container \"e25b2f97cf00d428e00e69355a3e04916764fd7a470202328c7df83322fd3600\": container with ID starting with e25b2f97cf00d428e00e69355a3e04916764fd7a470202328c7df83322fd3600 not found: ID does not exist"
Sep 16 17:55:48.954205 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:55:48.954147    2579 scope.go:117] "RemoveContainer" containerID="7668c0d762cdf82d93ee4657ef7eb2c2083fa519b549bfbc9dfa1f4441b25791"
Sep 16 17:55:48.954414 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:55:48.954397    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7668c0d762cdf82d93ee4657ef7eb2c2083fa519b549bfbc9dfa1f4441b25791\": container with ID starting with 7668c0d762cdf82d93ee4657ef7eb2c2083fa519b549bfbc9dfa1f4441b25791 not found: ID does not exist" containerID="7668c0d762cdf82d93ee4657ef7eb2c2083fa519b549bfbc9dfa1f4441b25791"
Sep 16 17:55:48.954476 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:55:48.954417    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7668c0d762cdf82d93ee4657ef7eb2c2083fa519b549bfbc9dfa1f4441b25791"} err="failed to get container status \"7668c0d762cdf82d93ee4657ef7eb2c2083fa519b549bfbc9dfa1f4441b25791\": rpc error: code = NotFound desc = could not find container \"7668c0d762cdf82d93ee4657ef7eb2c2083fa519b549bfbc9dfa1f4441b25791\": container with ID starting with 7668c0d762cdf82d93ee4657ef7eb2c2083fa519b549bfbc9dfa1f4441b25791 not found: ID does not exist"
Sep 16 17:55:48.954476 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:55:48.954431    2579 scope.go:117] "RemoveContainer" containerID="2d72af630da3349a406b2b456e52e479ffa156780f17b4034e388dc9095a2a32"
Sep 16 17:55:48.954676 ip-10-0-138-73 kubenswrapper[2579]: E0916 17:55:48.954658    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d72af630da3349a406b2b456e52e479ffa156780f17b4034e388dc9095a2a32\": container with ID starting with 2d72af630da3349a406b2b456e52e479ffa156780f17b4034e388dc9095a2a32 not found: ID does not exist" containerID="2d72af630da3349a406b2b456e52e479ffa156780f17b4034e388dc9095a2a32"
Sep 16 17:55:48.954750 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:55:48.954677    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-577b6fff5f-9whgv"]
Sep 16 17:55:48.954750 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:55:48.954680    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d72af630da3349a406b2b456e52e479ffa156780f17b4034e388dc9095a2a32"} err="failed to get container status \"2d72af630da3349a406b2b456e52e479ffa156780f17b4034e388dc9095a2a32\": rpc error: code = NotFound desc = could not find container \"2d72af630da3349a406b2b456e52e479ffa156780f17b4034e388dc9095a2a32\": container with ID starting with 2d72af630da3349a406b2b456e52e479ffa156780f17b4034e388dc9095a2a32 not found: ID does not exist"
Sep 16 17:55:48.963292 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:55:48.963266    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-577b6fff5f-9whgv"]
Sep 16 17:55:50.099170 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:55:50.099129    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="90f138c9-d05c-4dc5-b562-d3602188a542" path="/var/lib/kubelet/pods/90f138c9-d05c-4dc5-b562-d3602188a542/volumes"
Sep 16 17:55:50.690098 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:55:50.690062    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-znfk5/must-gather-n64zk"]
Sep 16 17:55:50.690431 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:55:50.690418    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="90f138c9-d05c-4dc5-b562-d3602188a542" containerName="kube-rbac-proxy"
Sep 16 17:55:50.690471 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:55:50.690434    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="90f138c9-d05c-4dc5-b562-d3602188a542" containerName="kube-rbac-proxy"
Sep 16 17:55:50.690471 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:55:50.690443    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cec9bb14-5925-4d45-8b99-308423d7df49" containerName="kube-rbac-proxy"
Sep 16 17:55:50.690471 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:55:50.690449    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="cec9bb14-5925-4d45-8b99-308423d7df49" containerName="kube-rbac-proxy"
Sep 16 17:55:50.690471 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:55:50.690459    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cec9bb14-5925-4d45-8b99-308423d7df49" containerName="kserve-container"
Sep 16 17:55:50.690471 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:55:50.690464    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="cec9bb14-5925-4d45-8b99-308423d7df49" containerName="kserve-container"
Sep 16 17:55:50.690631 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:55:50.690477    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="90f138c9-d05c-4dc5-b562-d3602188a542" containerName="agent"
Sep 16 17:55:50.690631 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:55:50.690483    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="90f138c9-d05c-4dc5-b562-d3602188a542" containerName="agent"
Sep 16 17:55:50.690631 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:55:50.690491    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="90f138c9-d05c-4dc5-b562-d3602188a542" containerName="storage-initializer"
Sep 16 17:55:50.690631 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:55:50.690496    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="90f138c9-d05c-4dc5-b562-d3602188a542" containerName="storage-initializer"
Sep 16 17:55:50.690631 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:55:50.690502    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="90f138c9-d05c-4dc5-b562-d3602188a542" containerName="kserve-container"
Sep 16 17:55:50.690631 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:55:50.690508    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="90f138c9-d05c-4dc5-b562-d3602188a542" containerName="kserve-container"
Sep 16 17:55:50.690631 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:55:50.690564    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="cec9bb14-5925-4d45-8b99-308423d7df49" containerName="kube-rbac-proxy"
Sep 16 17:55:50.690631 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:55:50.690572    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="cec9bb14-5925-4d45-8b99-308423d7df49" containerName="kserve-container"
Sep 16 17:55:50.690631 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:55:50.690579    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="90f138c9-d05c-4dc5-b562-d3602188a542" containerName="kserve-container"
Sep 16 17:55:50.690631 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:55:50.690586    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="90f138c9-d05c-4dc5-b562-d3602188a542" containerName="kube-rbac-proxy"
Sep 16 17:55:50.690631 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:55:50.690592    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="90f138c9-d05c-4dc5-b562-d3602188a542" containerName="agent"
Sep 16 17:55:50.693623 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:55:50.693606    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-znfk5/must-gather-n64zk"
Sep 16 17:55:50.697909 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:55:50.697888    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-znfk5\"/\"default-dockercfg-bh4xb\""
Sep 16 17:55:50.698064 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:55:50.697908    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-znfk5\"/\"kube-root-ca.crt\""
Sep 16 17:55:50.698341 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:55:50.698323    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-znfk5\"/\"openshift-service-ca.crt\""
Sep 16 17:55:50.710424 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:55:50.710392    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-znfk5/must-gather-n64zk"]
Sep 16 17:55:50.821971 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:55:50.821928    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7lkvc\" (UniqueName: \"kubernetes.io/projected/94a715c6-1afb-4afd-95f0-f1fabb764608-kube-api-access-7lkvc\") pod \"must-gather-n64zk\" (UID: \"94a715c6-1afb-4afd-95f0-f1fabb764608\") " pod="openshift-must-gather-znfk5/must-gather-n64zk"
Sep 16 17:55:50.822166 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:55:50.822012    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/94a715c6-1afb-4afd-95f0-f1fabb764608-must-gather-output\") pod \"must-gather-n64zk\" (UID: \"94a715c6-1afb-4afd-95f0-f1fabb764608\") " pod="openshift-must-gather-znfk5/must-gather-n64zk"
Sep 16 17:55:50.923293 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:55:50.923235    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/94a715c6-1afb-4afd-95f0-f1fabb764608-must-gather-output\") pod \"must-gather-n64zk\" (UID: \"94a715c6-1afb-4afd-95f0-f1fabb764608\") " pod="openshift-must-gather-znfk5/must-gather-n64zk"
Sep 16 17:55:50.923489 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:55:50.923327    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lkvc\" (UniqueName: \"kubernetes.io/projected/94a715c6-1afb-4afd-95f0-f1fabb764608-kube-api-access-7lkvc\") pod \"must-gather-n64zk\" (UID: \"94a715c6-1afb-4afd-95f0-f1fabb764608\") " pod="openshift-must-gather-znfk5/must-gather-n64zk"
Sep 16 17:55:50.923618 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:55:50.923597    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/94a715c6-1afb-4afd-95f0-f1fabb764608-must-gather-output\") pod \"must-gather-n64zk\" (UID: \"94a715c6-1afb-4afd-95f0-f1fabb764608\") " pod="openshift-must-gather-znfk5/must-gather-n64zk"
Sep 16 17:55:50.933594 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:55:50.933563    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7lkvc\" (UniqueName: \"kubernetes.io/projected/94a715c6-1afb-4afd-95f0-f1fabb764608-kube-api-access-7lkvc\") pod \"must-gather-n64zk\" (UID: \"94a715c6-1afb-4afd-95f0-f1fabb764608\") " pod="openshift-must-gather-znfk5/must-gather-n64zk"
Sep 16 17:55:51.014738 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:55:51.014647    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-znfk5/must-gather-n64zk"
Sep 16 17:55:51.297838 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:55:51.297804    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-znfk5/must-gather-n64zk"]
Sep 16 17:55:51.299356 ip-10-0-138-73 kubenswrapper[2579]: W0916 17:55:51.299331    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod94a715c6_1afb_4afd_95f0_f1fabb764608.slice/crio-c4bf2a968e01fe6a4e038e302b06bf463cafc42bfb62b3e6f0b631a16f650655 WatchSource:0}: Error finding container c4bf2a968e01fe6a4e038e302b06bf463cafc42bfb62b3e6f0b631a16f650655: Status 404 returned error can't find the container with id c4bf2a968e01fe6a4e038e302b06bf463cafc42bfb62b3e6f0b631a16f650655
Sep 16 17:55:51.301593 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:55:51.301577    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 17:55:51.939777 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:55:51.939737    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-znfk5/must-gather-n64zk" event={"ID":"94a715c6-1afb-4afd-95f0-f1fabb764608","Type":"ContainerStarted","Data":"c4bf2a968e01fe6a4e038e302b06bf463cafc42bfb62b3e6f0b631a16f650655"}
Sep 16 17:55:52.946866 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:55:52.946819    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-znfk5/must-gather-n64zk" event={"ID":"94a715c6-1afb-4afd-95f0-f1fabb764608","Type":"ContainerStarted","Data":"9e7391ea693c2eb6f9df11b355109ae92bdba298ba0a8fe0946153c6892b69dc"}
Sep 16 17:55:52.946866 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:55:52.946871    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-znfk5/must-gather-n64zk" event={"ID":"94a715c6-1afb-4afd-95f0-f1fabb764608","Type":"ContainerStarted","Data":"4c2435ebb9e5194948f51196156e22da6109928b69e4db65745d863e2b873180"}
Sep 16 17:55:52.972898 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:55:52.972818    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-znfk5/must-gather-n64zk" podStartSLOduration=2.215849251 podStartE2EDuration="2.972796901s" totalImagesPullingTime="756.94765ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:55:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:55:52.970096909 +0000 UTC m=+1865.440095056" watchObservedRunningTime="2026-09-16 17:55:52.972796901 +0000 UTC m=+1865.442795040"
Sep 16 17:55:53.784901 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:55:53.784868    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-s7zg6_e9240885-14d4-4b43-bf98-442e30d61c36/global-pull-secret-syncer/0.log"
Sep 16 17:55:53.982031 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:55:53.982002    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-gntnz_ef27aa30-cb45-45bc-8d0e-50107fc6459d/konnectivity-agent/0.log"
Sep 16 17:55:54.186671 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:55:54.186589    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-138-73.ec2.internal_ffbb989cf9bf70887695397b1f4835f5/haproxy/0.log"
Sep 16 17:55:57.894797 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:55:57.894768    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a0504e8b-dfaf-4e78-a021-cecd3ce94cac/alertmanager/0.log"
Sep 16 17:55:57.927132 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:55:57.927103    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a0504e8b-dfaf-4e78-a021-cecd3ce94cac/config-reloader/0.log"
Sep 16 17:55:57.960311 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:55:57.960274    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a0504e8b-dfaf-4e78-a021-cecd3ce94cac/kube-rbac-proxy-web/0.log"
Sep 16 17:55:57.990922 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:55:57.990896    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a0504e8b-dfaf-4e78-a021-cecd3ce94cac/kube-rbac-proxy/0.log"
Sep 16 17:55:58.017266 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:55:58.017231    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a0504e8b-dfaf-4e78-a021-cecd3ce94cac/kube-rbac-proxy-metric/0.log"
Sep 16 17:55:58.047935 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:55:58.047912    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a0504e8b-dfaf-4e78-a021-cecd3ce94cac/prom-label-proxy/0.log"
Sep 16 17:55:58.080372 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:55:58.080332    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a0504e8b-dfaf-4e78-a021-cecd3ce94cac/init-config-reloader/0.log"
Sep 16 17:55:58.132233 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:55:58.132191    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6cd954b897-8v98r_a37645e3-e094-46da-992b-c0ea722bc83f/cluster-monitoring-operator/0.log"
Sep 16 17:55:58.264746 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:55:58.264646    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-5b8b89bf95-slsg6_2a7ca0b4-ec03-407e-bade-0a533e7c1b15/metrics-server/0.log"
Sep 16 17:55:58.307345 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:55:58.307306    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5d85778b9d-d2fsj_5a36f055-8d5d-4bda-9a14-019707863d03/monitoring-plugin/0.log"
Sep 16 17:55:58.600889 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:55:58.600857    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-fbrpx_f8ebccfe-c8a8-4a90-8cad-c764e3ee9956/node-exporter/0.log"
Sep 16 17:55:58.629302 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:55:58.629276    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-fbrpx_f8ebccfe-c8a8-4a90-8cad-c764e3ee9956/kube-rbac-proxy/0.log"
Sep 16 17:55:58.660914 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:55:58.660877    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-fbrpx_f8ebccfe-c8a8-4a90-8cad-c764e3ee9956/init-textfile/0.log"
Sep 16 17:55:59.462285 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:55:59.462253    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-5864cd99d7-hw4m5_21c42a22-b64c-4528-bd63-d5d583ba59ab/prometheus-operator-admission-webhook/0.log"
Sep 16 17:55:59.527040 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:55:59.527008    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6d7f75957b-q998k_9ee7983c-145e-4621-b51c-2083d7e20177/telemeter-client/0.log"
Sep 16 17:55:59.570864 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:55:59.570815    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6d7f75957b-q998k_9ee7983c-145e-4621-b51c-2083d7e20177/reload/0.log"
Sep 16 17:55:59.621017 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:55:59.620982    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6d7f75957b-q998k_9ee7983c-145e-4621-b51c-2083d7e20177/kube-rbac-proxy/0.log"
Sep 16 17:55:59.697891 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:55:59.696042    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-654695c99f-4jmw9_27605ac2-4ce2-4847-b391-7f803ed3f809/thanos-query/0.log"
Sep 16 17:55:59.825658 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:55:59.825626    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-654695c99f-4jmw9_27605ac2-4ce2-4847-b391-7f803ed3f809/kube-rbac-proxy-web/0.log"
Sep 16 17:55:59.927873 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:55:59.927821    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-654695c99f-4jmw9_27605ac2-4ce2-4847-b391-7f803ed3f809/kube-rbac-proxy/0.log"
Sep 16 17:55:59.986461 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:55:59.986431    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-654695c99f-4jmw9_27605ac2-4ce2-4847-b391-7f803ed3f809/prom-label-proxy/0.log"
Sep 16 17:56:00.048552 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:56:00.048521    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-654695c99f-4jmw9_27605ac2-4ce2-4847-b391-7f803ed3f809/kube-rbac-proxy-rules/0.log"
Sep 16 17:56:00.110509 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:56:00.110437    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-654695c99f-4jmw9_27605ac2-4ce2-4847-b391-7f803ed3f809/kube-rbac-proxy-metrics/0.log"
Sep 16 17:56:01.626274 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:56:01.626229    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-znfk5/perf-node-gather-daemonset-hv6bt"]
Sep 16 17:56:01.632579 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:56:01.632547    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-znfk5/perf-node-gather-daemonset-hv6bt"
Sep 16 17:56:01.645700 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:56:01.645671    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-znfk5/perf-node-gather-daemonset-hv6bt"]
Sep 16 17:56:01.724639 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:56:01.724603    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zhl6\" (UniqueName: \"kubernetes.io/projected/a8d7fe6f-9268-43fd-8f7a-e26ba65c6f0e-kube-api-access-8zhl6\") pod \"perf-node-gather-daemonset-hv6bt\" (UID: \"a8d7fe6f-9268-43fd-8f7a-e26ba65c6f0e\") " pod="openshift-must-gather-znfk5/perf-node-gather-daemonset-hv6bt"
Sep 16 17:56:01.724803 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:56:01.724649    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a8d7fe6f-9268-43fd-8f7a-e26ba65c6f0e-lib-modules\") pod \"perf-node-gather-daemonset-hv6bt\" (UID: \"a8d7fe6f-9268-43fd-8f7a-e26ba65c6f0e\") " pod="openshift-must-gather-znfk5/perf-node-gather-daemonset-hv6bt"
Sep 16 17:56:01.724803 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:56:01.724685    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a8d7fe6f-9268-43fd-8f7a-e26ba65c6f0e-podres\") pod \"perf-node-gather-daemonset-hv6bt\" (UID: \"a8d7fe6f-9268-43fd-8f7a-e26ba65c6f0e\") " pod="openshift-must-gather-znfk5/perf-node-gather-daemonset-hv6bt"
Sep 16 17:56:01.724803 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:56:01.724775    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a8d7fe6f-9268-43fd-8f7a-e26ba65c6f0e-proc\") pod \"perf-node-gather-daemonset-hv6bt\" (UID: \"a8d7fe6f-9268-43fd-8f7a-e26ba65c6f0e\") " pod="openshift-must-gather-znfk5/perf-node-gather-daemonset-hv6bt"
Sep 16 17:56:01.724932 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:56:01.724899    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a8d7fe6f-9268-43fd-8f7a-e26ba65c6f0e-sys\") pod \"perf-node-gather-daemonset-hv6bt\" (UID: \"a8d7fe6f-9268-43fd-8f7a-e26ba65c6f0e\") " pod="openshift-must-gather-znfk5/perf-node-gather-daemonset-hv6bt"
Sep 16 17:56:01.826451 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:56:01.826414    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a8d7fe6f-9268-43fd-8f7a-e26ba65c6f0e-proc\") pod \"perf-node-gather-daemonset-hv6bt\" (UID: \"a8d7fe6f-9268-43fd-8f7a-e26ba65c6f0e\") " pod="openshift-must-gather-znfk5/perf-node-gather-daemonset-hv6bt"
Sep 16 17:56:01.826676 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:56:01.826556    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a8d7fe6f-9268-43fd-8f7a-e26ba65c6f0e-proc\") pod \"perf-node-gather-daemonset-hv6bt\" (UID: \"a8d7fe6f-9268-43fd-8f7a-e26ba65c6f0e\") " pod="openshift-must-gather-znfk5/perf-node-gather-daemonset-hv6bt"
Sep 16 17:56:01.826776 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:56:01.826649    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a8d7fe6f-9268-43fd-8f7a-e26ba65c6f0e-sys\") pod \"perf-node-gather-daemonset-hv6bt\" (UID: \"a8d7fe6f-9268-43fd-8f7a-e26ba65c6f0e\") " pod="openshift-must-gather-znfk5/perf-node-gather-daemonset-hv6bt"
Sep 16 17:56:01.826776 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:56:01.826701    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a8d7fe6f-9268-43fd-8f7a-e26ba65c6f0e-sys\") pod \"perf-node-gather-daemonset-hv6bt\" (UID: \"a8d7fe6f-9268-43fd-8f7a-e26ba65c6f0e\") " pod="openshift-must-gather-znfk5/perf-node-gather-daemonset-hv6bt"
Sep 16 17:56:01.826878 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:56:01.826773    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zhl6\" (UniqueName: \"kubernetes.io/projected/a8d7fe6f-9268-43fd-8f7a-e26ba65c6f0e-kube-api-access-8zhl6\") pod \"perf-node-gather-daemonset-hv6bt\" (UID: \"a8d7fe6f-9268-43fd-8f7a-e26ba65c6f0e\") " pod="openshift-must-gather-znfk5/perf-node-gather-daemonset-hv6bt"
Sep 16 17:56:01.826878 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:56:01.826804    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a8d7fe6f-9268-43fd-8f7a-e26ba65c6f0e-lib-modules\") pod \"perf-node-gather-daemonset-hv6bt\" (UID: \"a8d7fe6f-9268-43fd-8f7a-e26ba65c6f0e\") " pod="openshift-must-gather-znfk5/perf-node-gather-daemonset-hv6bt"
Sep 16 17:56:01.826878 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:56:01.826838    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a8d7fe6f-9268-43fd-8f7a-e26ba65c6f0e-podres\") pod \"perf-node-gather-daemonset-hv6bt\" (UID: \"a8d7fe6f-9268-43fd-8f7a-e26ba65c6f0e\") " pod="openshift-must-gather-znfk5/perf-node-gather-daemonset-hv6bt"
Sep 16 17:56:01.827035 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:56:01.826979    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a8d7fe6f-9268-43fd-8f7a-e26ba65c6f0e-podres\") pod \"perf-node-gather-daemonset-hv6bt\" (UID: \"a8d7fe6f-9268-43fd-8f7a-e26ba65c6f0e\") " pod="openshift-must-gather-znfk5/perf-node-gather-daemonset-hv6bt"
Sep 16 17:56:01.827035 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:56:01.826981    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a8d7fe6f-9268-43fd-8f7a-e26ba65c6f0e-lib-modules\") pod \"perf-node-gather-daemonset-hv6bt\" (UID: \"a8d7fe6f-9268-43fd-8f7a-e26ba65c6f0e\") " pod="openshift-must-gather-znfk5/perf-node-gather-daemonset-hv6bt"
Sep 16 17:56:01.835753 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:56:01.835713    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zhl6\" (UniqueName: \"kubernetes.io/projected/a8d7fe6f-9268-43fd-8f7a-e26ba65c6f0e-kube-api-access-8zhl6\") pod \"perf-node-gather-daemonset-hv6bt\" (UID: \"a8d7fe6f-9268-43fd-8f7a-e26ba65c6f0e\") " pod="openshift-must-gather-znfk5/perf-node-gather-daemonset-hv6bt"
Sep 16 17:56:01.947618 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:56:01.947515    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-znfk5/perf-node-gather-daemonset-hv6bt"
Sep 16 17:56:02.236922 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:56:02.236892    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-znfk5/perf-node-gather-daemonset-hv6bt"]
Sep 16 17:56:02.291578 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:56:02.291551    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-7784ccdc77-wdvpr_30e17757-071e-4e55-a611-89f630626cd2/console/0.log"
Sep 16 17:56:02.325592 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:56:02.325566    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-76f99584f4-xf777_ff8acb9f-034a-4aab-aed4-aebb3df42f07/download-server/0.log"
Sep 16 17:56:03.013560 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:56:03.013520    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-znfk5/perf-node-gather-daemonset-hv6bt" event={"ID":"a8d7fe6f-9268-43fd-8f7a-e26ba65c6f0e","Type":"ContainerStarted","Data":"5144d8c6c3699d454e4140550b32e27a045cebd89c60537762814bdf5170079e"}
Sep 16 17:56:03.014038 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:56:03.013582    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-znfk5/perf-node-gather-daemonset-hv6bt" event={"ID":"a8d7fe6f-9268-43fd-8f7a-e26ba65c6f0e","Type":"ContainerStarted","Data":"f094d6aa8bb7112195e212caa314976073f8706e471607f76ea87288c9c07936"}
Sep 16 17:56:03.014038 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:56:03.013699    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-znfk5/perf-node-gather-daemonset-hv6bt"
Sep 16 17:56:03.042218 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:56:03.042151    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-znfk5/perf-node-gather-daemonset-hv6bt" podStartSLOduration=2.042131042 podStartE2EDuration="2.042131042s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 17:56:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:56:03.03909522 +0000 UTC m=+1875.509093355" watchObservedRunningTime="2026-09-16 17:56:03.042131042 +0000 UTC m=+1875.512129190"
Sep 16 17:56:03.936428 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:56:03.936394    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-9cbqd_a639ab67-695c-4ec2-af20-1882ac49c4df/dns/0.log"
Sep 16 17:56:03.968580 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:56:03.968547    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-9cbqd_a639ab67-695c-4ec2-af20-1882ac49c4df/kube-rbac-proxy/0.log"
Sep 16 17:56:04.251737 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:56:04.251656    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-qtdvz_21a2db47-94e4-4d2b-9cb2-0b2bca64ef5b/dns-node-resolver/0.log"
Sep 16 17:56:04.971410 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:56:04.971380    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-t6cx6_41b48733-fd0e-4eaa-a91c-5d57f642b2f8/node-ca/0.log"
Sep 16 17:56:06.090752 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:56:06.090722    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-99d9f4-g4nhp_7a9b26b4-3f5f-40d2-b866-ad3915e771bb/router/0.log"
Sep 16 17:56:06.567165 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:56:06.567135    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-t8qph_7886f4b3-cb86-4a0e-ae6c-809ac7a567fa/serve-healthcheck-canary/0.log"
Sep 16 17:56:07.191622 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:56:07.191592    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-zhxvb_3b4699f3-b8d0-408c-9031-37f1ec7cd316/kube-rbac-proxy/0.log"
Sep 16 17:56:07.215454 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:56:07.215427    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-zhxvb_3b4699f3-b8d0-408c-9031-37f1ec7cd316/exporter/0.log"
Sep 16 17:56:07.239155 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:56:07.239129    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-zhxvb_3b4699f3-b8d0-408c-9031-37f1ec7cd316/extractor/0.log"
Sep 16 17:56:09.027795 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:56:09.027766    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-znfk5/perf-node-gather-daemonset-hv6bt"
Sep 16 17:56:09.620163 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:56:09.620133    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_kserve-controller-manager-cc8594966-hz8vc_243a73bb-a95e-4c0d-8fda-c2d3a1236b9e/manager/0.log"
Sep 16 17:56:09.711161 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:56:09.711129    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_model-serving-api-7b954ffc95-dcwxf_f07f0722-dbc1-441a-8d3a-5b560f1d5e12/server/0.log"
Sep 16 17:56:09.852668 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:56:09.852631    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-7b966d7984-wc25q_8a6ddf5b-ecfa-4bf7-9acc-a2e96c17676c/manager/0.log"
Sep 16 17:56:09.951018 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:56:09.950925    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-64568bcd49-gzwdn_a9dc8c86-0e70-401e-b73e-65e7d9768032/seaweedfs/0.log"
Sep 16 17:56:14.743107 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:56:14.743071    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-8b4d697c5-vdcxh_4a2652f6-34d3-4fe8-82ec-2501aa923133/migrator/0.log"
Sep 16 17:56:14.803255 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:56:14.803228    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-8b4d697c5-vdcxh_4a2652f6-34d3-4fe8-82ec-2501aa923133/graceful-termination/0.log"
Sep 16 17:56:15.327597 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:56:15.327567    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6c8f84f897-f7kcw_51b86650-5910-4bc0-9832-dc967737cbea/kube-storage-version-migrator-operator/1.log"
Sep 16 17:56:15.328476 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:56:15.328455    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6c8f84f897-f7kcw_51b86650-5910-4bc0-9832-dc967737cbea/kube-storage-version-migrator-operator/0.log"
Sep 16 17:56:16.514751 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:56:16.514720    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-59rw5_a1223407-0c9e-4ed6-bd17-cf0f3962e27e/kube-multus/0.log"
Sep 16 17:56:16.599510 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:56:16.599476    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6fsn7_5df1fb59-c460-463c-8691-2c270b69aa34/kube-multus-additional-cni-plugins/0.log"
Sep 16 17:56:16.629171 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:56:16.629123    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6fsn7_5df1fb59-c460-463c-8691-2c270b69aa34/egress-router-binary-copy/0.log"
Sep 16 17:56:16.663725 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:56:16.663672    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6fsn7_5df1fb59-c460-463c-8691-2c270b69aa34/cni-plugins/0.log"
Sep 16 17:56:16.697695 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:56:16.697667    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6fsn7_5df1fb59-c460-463c-8691-2c270b69aa34/bond-cni-plugin/0.log"
Sep 16 17:56:16.732331 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:56:16.732305    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6fsn7_5df1fb59-c460-463c-8691-2c270b69aa34/routeoverride-cni/0.log"
Sep 16 17:56:16.764461 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:56:16.764437    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6fsn7_5df1fb59-c460-463c-8691-2c270b69aa34/whereabouts-cni-bincopy/0.log"
Sep 16 17:56:16.821004 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:56:16.820971    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6fsn7_5df1fb59-c460-463c-8691-2c270b69aa34/whereabouts-cni/0.log"
Sep 16 17:56:17.710581 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:56:17.710555    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-xfgjq_bc719829-66b1-4a84-b994-e71cf111dafa/network-metrics-daemon/0.log"
Sep 16 17:56:17.737214 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:56:17.737191    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-xfgjq_bc719829-66b1-4a84-b994-e71cf111dafa/kube-rbac-proxy/0.log"
Sep 16 17:56:18.542397 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:56:18.542368    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-f9qxl_c2617cf7-6369-4fc4-9fb5-6e6cfc4e9add/ovn-controller/0.log"
Sep 16 17:56:18.581440 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:56:18.581411    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-f9qxl_c2617cf7-6369-4fc4-9fb5-6e6cfc4e9add/ovn-acl-logging/0.log"
Sep 16 17:56:18.616155 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:56:18.616061    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-f9qxl_c2617cf7-6369-4fc4-9fb5-6e6cfc4e9add/kube-rbac-proxy-node/0.log"
Sep 16 17:56:18.644139 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:56:18.644089    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-f9qxl_c2617cf7-6369-4fc4-9fb5-6e6cfc4e9add/kube-rbac-proxy-ovn-metrics/0.log"
Sep 16 17:56:18.666050 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:56:18.666021    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-f9qxl_c2617cf7-6369-4fc4-9fb5-6e6cfc4e9add/northd/0.log"
Sep 16 17:56:18.694310 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:56:18.694280    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-f9qxl_c2617cf7-6369-4fc4-9fb5-6e6cfc4e9add/nbdb/0.log"
Sep 16 17:56:18.725094 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:56:18.725060    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-f9qxl_c2617cf7-6369-4fc4-9fb5-6e6cfc4e9add/sbdb/0.log"
Sep 16 17:56:18.782499 ip-10-0-138-73 kubenswrapper[2579]: I0916 17:56:18.782460    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-f9qxl_c2617cf7-6369-4fc4-9fb5-6e6cfc4e9add/ovnkube-controller/0.log"