May 25 13:03:11.683826 ip-10-0-143-235 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
May 25 13:03:11.683838 ip-10-0-143-235 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
May 25 13:03:11.683848 ip-10-0-143-235 systemd[1]: kubelet.service: Failed with result 'resources'.
May 25 13:03:11.684154 ip-10-0-143-235 systemd[1]: Failed to start Kubernetes Kubelet.
May 25 13:03:21.911727 ip-10-0-143-235 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
May 25 13:03:21.911742 ip-10-0-143-235 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 6ea9f554861248d5a8faf43bc0deda0b --
May 25 13:05:47.408658 ip-10-0-143-235 systemd[1]: Starting Kubernetes Kubelet...
May 25 13:05:47.884596 ip-10-0-143-235 kubenswrapper[2557]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
May 25 13:05:47.884596 ip-10-0-143-235 kubenswrapper[2557]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
May 25 13:05:47.884596 ip-10-0-143-235 kubenswrapper[2557]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
May 25 13:05:47.884596 ip-10-0-143-235 kubenswrapper[2557]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
May 25 13:05:47.884596 ip-10-0-143-235 kubenswrapper[2557]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
May 25 13:05:47.887092 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:47.886990    2557 server.go:212] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
May 25 13:05:47.890756 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.890738    2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 25 13:05:47.890756 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.890755    2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 25 13:05:47.890821 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.890759    2557 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 25 13:05:47.890821 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.890763    2557 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 25 13:05:47.890821 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.890766    2557 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 25 13:05:47.890821 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.890770    2557 feature_gate.go:328] unrecognized feature gate: Example2
May 25 13:05:47.890821 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.890773    2557 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 25 13:05:47.890821 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.890775    2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 25 13:05:47.890821 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.890778    2557 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 25 13:05:47.890821 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.890780    2557 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 25 13:05:47.890821 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.890783    2557 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 25 13:05:47.890821 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.890785    2557 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 25 13:05:47.890821 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.890788    2557 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 25 13:05:47.890821 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.890791    2557 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 25 13:05:47.890821 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.890794    2557 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 25 13:05:47.890821 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.890796    2557 feature_gate.go:328] unrecognized feature gate: DualReplica
May 25 13:05:47.890821 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.890799    2557 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 25 13:05:47.890821 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.890802    2557 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 25 13:05:47.890821 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.890804    2557 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 25 13:05:47.890821 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.890807    2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 25 13:05:47.890821 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.890811    2557 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 25 13:05:47.890821 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.890814    2557 feature_gate.go:328] unrecognized feature gate: NewOLM
May 25 13:05:47.891292 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.890816    2557 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 25 13:05:47.891292 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.890819    2557 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 25 13:05:47.891292 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.890822    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 25 13:05:47.891292 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.890824    2557 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 25 13:05:47.891292 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.890827    2557 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 25 13:05:47.891292 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.890830    2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 25 13:05:47.891292 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.890833    2557 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 25 13:05:47.891292 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.890835    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 25 13:05:47.891292 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.890838    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 25 13:05:47.891292 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.890841    2557 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 25 13:05:47.891292 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.890843    2557 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 25 13:05:47.891292 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.890847    2557 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 25 13:05:47.891292 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.890849    2557 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 25 13:05:47.891292 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.890852    2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 25 13:05:47.891292 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.890854    2557 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 25 13:05:47.891292 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.890857    2557 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 25 13:05:47.891292 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.890859    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 25 13:05:47.891292 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.890861    2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 25 13:05:47.891292 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.890864    2557 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 25 13:05:47.891292 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.890866    2557 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 25 13:05:47.891832 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.890869    2557 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 25 13:05:47.891832 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.890872    2557 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 25 13:05:47.891832 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.890875    2557 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 25 13:05:47.891832 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.890877    2557 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 25 13:05:47.891832 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.890881    2557 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 25 13:05:47.891832 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.890886    2557 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 25 13:05:47.891832 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.890889    2557 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 25 13:05:47.891832 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.890892    2557 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 25 13:05:47.891832 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.890895    2557 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 25 13:05:47.891832 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.890898    2557 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 25 13:05:47.891832 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.890901    2557 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 25 13:05:47.891832 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.890904    2557 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 25 13:05:47.891832 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.890906    2557 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 25 13:05:47.891832 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.890910    2557 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 25 13:05:47.891832 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.890912    2557 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 25 13:05:47.891832 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.890915    2557 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 25 13:05:47.891832 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.890918    2557 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 25 13:05:47.891832 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.890920    2557 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 25 13:05:47.891832 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.890923    2557 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 25 13:05:47.892289 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.890926    2557 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 25 13:05:47.892289 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.890928    2557 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 25 13:05:47.892289 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.890931    2557 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 25 13:05:47.892289 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.890934    2557 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 25 13:05:47.892289 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.890936    2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 25 13:05:47.892289 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.890939    2557 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 25 13:05:47.892289 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.890942    2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 25 13:05:47.892289 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.890945    2557 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 25 13:05:47.892289 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.890947    2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 25 13:05:47.892289 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.890950    2557 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 25 13:05:47.892289 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.890952    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 25 13:05:47.892289 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.890958    2557 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 25 13:05:47.892289 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.890962    2557 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 25 13:05:47.892289 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.890965    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 25 13:05:47.892289 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.890968    2557 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 25 13:05:47.892289 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.890971    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 25 13:05:47.892289 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.890973    2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 25 13:05:47.892289 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.890976    2557 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 25 13:05:47.892289 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.890979    2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 25 13:05:47.892789 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.890982    2557 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 25 13:05:47.892789 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.890985    2557 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 25 13:05:47.892789 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.890988    2557 feature_gate.go:328] unrecognized feature gate: Example
May 25 13:05:47.892789 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.890991    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 25 13:05:47.892789 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.890993    2557 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 25 13:05:47.892789 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.890996    2557 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 25 13:05:47.892789 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.891426    2557 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 25 13:05:47.892789 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.891433    2557 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 25 13:05:47.892789 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.891436    2557 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 25 13:05:47.892789 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.891439    2557 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 25 13:05:47.892789 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.891442    2557 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 25 13:05:47.892789 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.891445    2557 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 25 13:05:47.892789 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.891448    2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 25 13:05:47.892789 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.891451    2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 25 13:05:47.892789 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.891455    2557 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 25 13:05:47.892789 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.891458    2557 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 25 13:05:47.892789 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.891461    2557 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 25 13:05:47.892789 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.891464    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 25 13:05:47.892789 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.891466    2557 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 25 13:05:47.892789 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.891469    2557 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 25 13:05:47.893262 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.891472    2557 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 25 13:05:47.893262 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.891474    2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 25 13:05:47.893262 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.891477    2557 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 25 13:05:47.893262 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.891479    2557 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 25 13:05:47.893262 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.891483    2557 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 25 13:05:47.893262 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.891485    2557 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 25 13:05:47.893262 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.891488    2557 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 25 13:05:47.893262 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.891490    2557 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 25 13:05:47.893262 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.891493    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 25 13:05:47.893262 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.891495    2557 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 25 13:05:47.893262 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.891498    2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 25 13:05:47.893262 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.891501    2557 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 25 13:05:47.893262 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.891503    2557 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 25 13:05:47.893262 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.891506    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 25 13:05:47.893262 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.891509    2557 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 25 13:05:47.893262 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.891511    2557 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 25 13:05:47.893262 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.891514    2557 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 25 13:05:47.893262 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.891517    2557 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 25 13:05:47.893262 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.891520    2557 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 25 13:05:47.893262 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.891524    2557 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 25 13:05:47.893771 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.891526    2557 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 25 13:05:47.893771 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.891529    2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 25 13:05:47.893771 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.891531    2557 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 25 13:05:47.893771 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.891534    2557 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 25 13:05:47.893771 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.891536    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 25 13:05:47.893771 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.891539    2557 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 25 13:05:47.893771 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.891542    2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 25 13:05:47.893771 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.891545    2557 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 25 13:05:47.893771 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.891547    2557 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 25 13:05:47.893771 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.891549    2557 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 25 13:05:47.893771 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.891552    2557 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 25 13:05:47.893771 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.891554    2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 25 13:05:47.893771 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.891557    2557 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 25 13:05:47.893771 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.891559    2557 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 25 13:05:47.893771 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.891562    2557 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 25 13:05:47.893771 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.891564    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 25 13:05:47.893771 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.891567    2557 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 25 13:05:47.893771 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.891570    2557 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 25 13:05:47.893771 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.891573    2557 feature_gate.go:328] unrecognized feature gate: Example2
May 25 13:05:47.894287 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.891575    2557 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 25 13:05:47.894287 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.891577    2557 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 25 13:05:47.894287 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.891580    2557 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 25 13:05:47.894287 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.891583    2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 25 13:05:47.894287 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.891585    2557 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 25 13:05:47.894287 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.891588    2557 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 25 13:05:47.894287 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.891590    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 25 13:05:47.894287 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.891593    2557 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 25 13:05:47.894287 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.891596    2557 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 25 13:05:47.894287 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.891598    2557 feature_gate.go:328] unrecognized feature gate: NewOLM
May 25 13:05:47.894287 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.891601    2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 25 13:05:47.894287 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.891604    2557 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 25 13:05:47.894287 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.891607    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 25 13:05:47.894287 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.891609    2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 25 13:05:47.894287 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.891612    2557 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 25 13:05:47.894287 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.891615    2557 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 25 13:05:47.894287 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.891617    2557 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 25 13:05:47.894287 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.891620    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 25 13:05:47.894287 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.891622    2557 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 25 13:05:47.894287 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.891625    2557 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 25 13:05:47.894812 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.891628    2557 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 25 13:05:47.894812 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.891630    2557 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 25 13:05:47.894812 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.891632    2557 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 25 13:05:47.894812 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.891635    2557 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 25 13:05:47.894812 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.891637    2557 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 25 13:05:47.894812 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.891640    2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 25 13:05:47.894812 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.891642    2557 feature_gate.go:328] unrecognized feature gate: Example
May 25 13:05:47.894812 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.891645    2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 25 13:05:47.894812 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.891648    2557 feature_gate.go:328] unrecognized feature gate: DualReplica
May 25 13:05:47.894812 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.891651    2557 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 25 13:05:47.894812 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.891655    2557 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 25 13:05:47.894812 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.891659    2557 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 25 13:05:47.894812 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.891662    2557 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 25 13:05:47.894812 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:47.892882    2557 flags.go:64] FLAG: --address="0.0.0.0"
May 25 13:05:47.894812 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:47.892893    2557 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
May 25 13:05:47.894812 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:47.892899    2557 flags.go:64] FLAG: --anonymous-auth="true"
May 25 13:05:47.894812 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:47.892904    2557 flags.go:64] FLAG: --application-metrics-count-limit="100"
May 25 13:05:47.894812 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:47.892909    2557 flags.go:64] FLAG: --authentication-token-webhook="false"
May 25 13:05:47.894812 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:47.892912    2557 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
May 25 13:05:47.894812 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:47.892917    2557 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
May 25 13:05:47.894812 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:47.892922    2557 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
May 25 13:05:47.895326 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:47.892926    2557 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
May 25 13:05:47.895326 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:47.892930    2557 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
May 25 13:05:47.895326 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:47.892934    2557 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
May 25 13:05:47.895326 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:47.892938    2557 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
May 25 13:05:47.895326 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:47.892941    2557 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
May 25 13:05:47.895326 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:47.892944    2557 flags.go:64] FLAG: --cgroup-root=""
May 25 13:05:47.895326 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:47.892947    2557 flags.go:64] FLAG: --cgroups-per-qos="true"
May 25 13:05:47.895326 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:47.892951    2557 flags.go:64] FLAG: --client-ca-file=""
May 25 13:05:47.895326 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:47.892954    2557 flags.go:64] FLAG: --cloud-config=""
May 25 13:05:47.895326 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:47.892957    2557 flags.go:64] FLAG: --cloud-provider="external"
May 25 13:05:47.895326 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:47.892959    2557 flags.go:64] FLAG: --cluster-dns="[]"
May 25 13:05:47.895326 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:47.892965    2557 flags.go:64] FLAG: --cluster-domain=""
May 25 13:05:47.895326 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:47.892968    2557 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
May 25 13:05:47.895326 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:47.892971    2557 flags.go:64] FLAG: --config-dir=""
May 25 13:05:47.895326 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:47.892975    2557 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
May 25 13:05:47.895326 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:47.892978    2557 flags.go:64] FLAG: --container-log-max-files="5"
May 25 13:05:47.895326 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:47.892983    2557 flags.go:64] FLAG: --container-log-max-size="10Mi"
May 25 13:05:47.895326 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:47.892986    2557 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
May 25 13:05:47.895326 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:47.892989    2557 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
May 25 13:05:47.895326 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:47.892992    2557 flags.go:64] FLAG: --containerd-namespace="k8s.io"
May 25 13:05:47.895326 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:47.892995    2557 flags.go:64] FLAG: --contention-profiling="false"
May 25 13:05:47.895326 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:47.892998    2557 flags.go:64] FLAG: --cpu-cfs-quota="true"
May 25 13:05:47.895326 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:47.893001    2557 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
May 25 13:05:47.895326 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:47.893005    2557 flags.go:64] FLAG: --cpu-manager-policy="none"
May 25 13:05:47.895326 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:47.893008    2557 flags.go:64] FLAG: --cpu-manager-policy-options=""
May 25 13:05:47.895946 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:47.893013    2557 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
May 25 13:05:47.895946 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:47.893016    2557 flags.go:64] FLAG: --enable-controller-attach-detach="true"
May 25 13:05:47.895946 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:47.893019    2557 flags.go:64] FLAG: --enable-debugging-handlers="true"
May 25 13:05:47.895946 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:47.893022    2557 flags.go:64] FLAG: --enable-load-reader="false"
May 25 13:05:47.895946 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:47.893025    2557 flags.go:64] FLAG: --enable-server="true"
May 25 13:05:47.895946 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:47.893028    2557 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
May 25 13:05:47.895946 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:47.893032    2557 flags.go:64] FLAG: --event-burst="100"
May 25 13:05:47.895946 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:47.893035    2557 flags.go:64] FLAG: --event-qps="50"
May 25 13:05:47.895946 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:47.893038    2557 flags.go:64] FLAG: --event-storage-age-limit="default=0"
May 25 13:05:47.895946 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:47.893042    2557 flags.go:64] FLAG: --event-storage-event-limit="default=0"
May 25 13:05:47.895946 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:47.893045    2557 flags.go:64] FLAG: --eviction-hard=""
May 25 13:05:47.895946 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:47.893049    2557 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
May 25 13:05:47.895946 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:47.893052    2557 flags.go:64] FLAG: --eviction-minimum-reclaim=""
May 25 13:05:47.895946 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:47.893056    2557 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
May 25 13:05:47.895946 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:47.893058    2557 flags.go:64] FLAG: --eviction-soft=""
May 25 13:05:47.895946 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:47.893062    2557 flags.go:64] FLAG: --eviction-soft-grace-period=""
May 25 13:05:47.895946 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:47.893065    2557 flags.go:64] FLAG: --exit-on-lock-contention="false"
May 25 13:05:47.895946 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:47.893067    2557 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
May 25 13:05:47.895946 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:47.893070    2557 flags.go:64] FLAG: --experimental-mounter-path=""
May 25 13:05:47.895946 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:47.893073    2557 flags.go:64] FLAG: --fail-cgroupv1="false"
May 25 13:05:47.895946 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:47.893076    2557 flags.go:64] FLAG: --fail-swap-on="true"
May 25 13:05:47.895946 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:47.893079    2557 flags.go:64] FLAG: --feature-gates=""
May 25 13:05:47.895946 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:47.893083    2557 flags.go:64] FLAG: --file-check-frequency="20s"
May 25 13:05:47.895946 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:47.893087    2557 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
May 25 13:05:47.895946 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:47.893089    2557 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
May 25 13:05:47.896562 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:47.893093    2557 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
May 25 13:05:47.896562 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:47.893096    2557 flags.go:64] FLAG: --healthz-port="10248"
May 25 13:05:47.896562 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:47.893099    2557 flags.go:64] FLAG: --help="false"
May 25 13:05:47.896562 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:47.893103    2557 flags.go:64] FLAG: --hostname-override="ip-10-0-143-235.ec2.internal"
May 25 13:05:47.896562 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:47.893106    2557 flags.go:64] FLAG: --housekeeping-interval="10s"
May 25 13:05:47.896562 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:47.893109    2557 flags.go:64] FLAG: --http-check-frequency="20s"
May 25 13:05:47.896562 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:47.893113    2557 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
May 25 13:05:47.896562 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:47.893117    2557 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
May 25 13:05:47.896562 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:47.893120    2557 flags.go:64] FLAG: --image-gc-high-threshold="85"
May 25 13:05:47.896562 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:47.893123    2557 flags.go:64] FLAG: --image-gc-low-threshold="80"
May 25 13:05:47.896562 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:47.893126    2557 flags.go:64] FLAG: --image-service-endpoint=""
May 25 13:05:47.896562 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:47.893129    2557 flags.go:64] FLAG: --kernel-memcg-notification="false"
May 25 13:05:47.896562 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:47.893132    2557 flags.go:64] FLAG: --kube-api-burst="100"
May 25 13:05:47.896562 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:47.893135    2557 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
May 25 13:05:47.896562 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:47.893138    2557 flags.go:64] FLAG: --kube-api-qps="50"
May 25 13:05:47.896562 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:47.893141    2557 flags.go:64] FLAG: --kube-reserved=""
May 25 13:05:47.896562 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:47.893144    2557 flags.go:64] FLAG: --kube-reserved-cgroup=""
May 25 13:05:47.896562 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:47.893147    2557 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
May 25 13:05:47.896562 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:47.893150    2557 flags.go:64] FLAG: --kubelet-cgroups=""
May 25 13:05:47.896562 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:47.893154    2557 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
May 25 13:05:47.896562 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:47.893157    2557 flags.go:64] FLAG: --lock-file=""
May 25 13:05:47.896562 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:47.893160    2557 flags.go:64] FLAG: --log-cadvisor-usage="false"
May 25 13:05:47.896562 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:47.893163    2557 flags.go:64] FLAG: --log-flush-frequency="5s"
May 25 13:05:47.896562 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:47.893166    2557 flags.go:64] FLAG: --log-json-info-buffer-size="0"
May 25 13:05:47.897138 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:47.893172    2557 flags.go:64] FLAG: --log-json-split-stream="false"
May 25 13:05:47.897138 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:47.893174    2557 flags.go:64] FLAG: --log-text-info-buffer-size="0"
May 25 13:05:47.897138 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:47.893177    2557 flags.go:64] FLAG: --log-text-split-stream="false"
May 25 13:05:47.897138 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:47.893181    2557 flags.go:64] FLAG: --logging-format="text"
May 25 13:05:47.897138 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:47.893184    2557 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
May 25 13:05:47.897138 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:47.893187    2557 flags.go:64] FLAG: --make-iptables-util-chains="true"
May 25 13:05:47.897138 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:47.893190    2557 flags.go:64] FLAG: --manifest-url=""
May 25 13:05:47.897138 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:47.893193    2557 flags.go:64] FLAG: --manifest-url-header=""
May 25 13:05:47.897138 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:47.893197    2557 flags.go:64] FLAG: --max-housekeeping-interval="15s"
May 25 13:05:47.897138 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:47.893200    2557 flags.go:64] FLAG: --max-open-files="1000000"
May 25 13:05:47.897138 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:47.893205    2557 flags.go:64] FLAG: --max-pods="110"
May 25 13:05:47.897138 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:47.893208    2557 flags.go:64] FLAG: --maximum-dead-containers="-1"
May 25 13:05:47.897138 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:47.893211    2557 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
May 25 13:05:47.897138 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:47.893214    2557 flags.go:64] FLAG: --memory-manager-policy="None"
May 25 13:05:47.897138 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:47.893217    2557 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
May 25 13:05:47.897138 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:47.893221    2557 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
May 25 13:05:47.897138 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:47.893224    2557 flags.go:64] FLAG: --node-ip="0.0.0.0"
May 25 13:05:47.897138 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:47.893227    2557 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
May 25 13:05:47.897138 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:47.893234    2557 flags.go:64] FLAG: --node-status-max-images="50"
May 25 13:05:47.897138 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:47.893238    2557 flags.go:64] FLAG: --node-status-update-frequency="10s"
May 25 13:05:47.897138 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:47.893241    2557 flags.go:64] FLAG: --oom-score-adj="-999"
May 25 13:05:47.897138 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:47.893244    2557 flags.go:64] FLAG: --pod-cidr=""
May 25 13:05:47.897138 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:47.893247    2557 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:afe88671bf163c42b467d159dc6f0b784dbe28b1ed44e171edc9c06fd99cc6fc"
May 25 13:05:47.897786 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:47.893253    2557 flags.go:64] FLAG: --pod-manifest-path=""
May 25 13:05:47.897786 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:47.893256    2557 flags.go:64] FLAG: --pod-max-pids="-1"
May 25 13:05:47.897786 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:47.893260    2557 flags.go:64] FLAG: --pods-per-core="0"
May 25 13:05:47.897786 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:47.893263    2557 flags.go:64] FLAG: --port="10250"
May 25 13:05:47.897786 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:47.893266    2557 flags.go:64] FLAG: --protect-kernel-defaults="false"
May 25 13:05:47.897786 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:47.893269    2557 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0f66b7109d47baacd"
May 25 13:05:47.897786 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:47.893272    2557 flags.go:64] FLAG: --qos-reserved=""
May 25 13:05:47.897786 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:47.893275    2557 flags.go:64] FLAG: --read-only-port="10255"
May 25 13:05:47.897786 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:47.893278    2557 flags.go:64] FLAG: --register-node="true"
May 25 13:05:47.897786 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:47.893281    2557 flags.go:64] FLAG: --register-schedulable="true"
May 25 13:05:47.897786 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:47.893284    2557 flags.go:64] FLAG: --register-with-taints=""
May 25 13:05:47.897786 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:47.893288    2557 flags.go:64] FLAG: --registry-burst="10"
May 25 13:05:47.897786 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:47.893291    2557 flags.go:64] FLAG: --registry-qps="5"
May 25 13:05:47.897786 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:47.893294    2557 flags.go:64] FLAG: --reserved-cpus=""
May 25 13:05:47.897786 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:47.893297    2557 flags.go:64] FLAG: --reserved-memory=""
May 25 13:05:47.897786 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:47.893301    2557 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
May 25 13:05:47.897786 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:47.893304    2557 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
May 25 13:05:47.897786 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:47.893308    2557 flags.go:64] FLAG: --rotate-certificates="false"
May 25 13:05:47.897786 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:47.893310    2557 flags.go:64] FLAG: --rotate-server-certificates="false"
May 25 13:05:47.897786 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:47.893313    2557 flags.go:64] FLAG: --runonce="false"
May 25 13:05:47.897786 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:47.893316    2557 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
May 25 13:05:47.897786 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:47.893319    2557 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
May 25 13:05:47.897786 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:47.893322    2557 flags.go:64] FLAG: --seccomp-default="false"
May 25 13:05:47.897786 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:47.893325    2557 flags.go:64] FLAG: --serialize-image-pulls="true"
May 25 13:05:47.897786 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:47.893328    2557 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
May 25 13:05:47.897786 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:47.893331    2557 flags.go:64] FLAG: --storage-driver-db="cadvisor"
May 25 13:05:47.898446 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:47.893335    2557 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
May 25 13:05:47.898446 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:47.893338    2557 flags.go:64] FLAG: --storage-driver-password="root"
May 25 13:05:47.898446 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:47.893341    2557 flags.go:64] FLAG: --storage-driver-secure="false"
May 25 13:05:47.898446 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:47.893355    2557 flags.go:64] FLAG: --storage-driver-table="stats"
May 25 13:05:47.898446 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:47.893358    2557 flags.go:64] FLAG: --storage-driver-user="root"
May 25 13:05:47.898446 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:47.893361    2557 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
May 25 13:05:47.898446 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:47.893364    2557 flags.go:64] FLAG: --sync-frequency="1m0s"
May 25 13:05:47.898446 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:47.893367    2557 flags.go:64] FLAG: --system-cgroups=""
May 25 13:05:47.898446 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:47.893370    2557 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
May 25 13:05:47.898446 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:47.893375    2557 flags.go:64] FLAG: --system-reserved-cgroup=""
May 25 13:05:47.898446 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:47.893378    2557 flags.go:64] FLAG: --tls-cert-file=""
May 25 13:05:47.898446 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:47.893381    2557 flags.go:64] FLAG: --tls-cipher-suites="[]"
May 25 13:05:47.898446 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:47.893386    2557 flags.go:64] FLAG: --tls-min-version=""
May 25 13:05:47.898446 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:47.893389    2557 flags.go:64] FLAG: --tls-private-key-file=""
May 25 13:05:47.898446 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:47.893391    2557 flags.go:64] FLAG: --topology-manager-policy="none"
May 25 13:05:47.898446 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:47.893394    2557 flags.go:64] FLAG: --topology-manager-policy-options=""
May 25 13:05:47.898446 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:47.893397    2557 flags.go:64] FLAG: --topology-manager-scope="container"
May 25 13:05:47.898446 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:47.893400    2557 flags.go:64] FLAG: --v="2"
May 25 13:05:47.898446 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:47.893404    2557 flags.go:64] FLAG: --version="false"
May 25 13:05:47.898446 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:47.893409    2557 flags.go:64] FLAG: --vmodule=""
May 25 13:05:47.898446 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:47.893413    2557 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
May 25 13:05:47.898446 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:47.893417    2557 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
May 25 13:05:47.898446 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.893520    2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 25 13:05:47.898446 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.893524    2557 feature_gate.go:328] unrecognized feature gate: Example2
May 25 13:05:47.898446 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.893527    2557 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 25 13:05:47.899052 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.893530    2557 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 25 13:05:47.899052 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.893532    2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 25 13:05:47.899052 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.893535    2557 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 25 13:05:47.899052 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.893538    2557 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 25 13:05:47.899052 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.893541    2557 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 25 13:05:47.899052 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.893543    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 25 13:05:47.899052 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.893546    2557 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 25 13:05:47.899052 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.893549    2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 25 13:05:47.899052 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.893552    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 25 13:05:47.899052 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.893555    2557 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 25 13:05:47.899052 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.893558    2557 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 25 13:05:47.899052 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.893560    2557 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 25 13:05:47.899052 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.893567    2557 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 25 13:05:47.899052 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.893570    2557 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 25 13:05:47.899052 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.893573    2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 25 13:05:47.899052 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.893576    2557 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 25 13:05:47.899052 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.893578    2557 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 25 13:05:47.899052 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.893581    2557 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 25 13:05:47.899052 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.893584    2557 feature_gate.go:328] unrecognized feature gate: NewOLM
May 25 13:05:47.899052 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.893587    2557 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 25 13:05:47.899575 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.893589    2557 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 25 13:05:47.899575 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.893592    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 25 13:05:47.899575 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.893595    2557 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 25 13:05:47.899575 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.893597    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 25 13:05:47.899575 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.893600    2557 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 25 13:05:47.899575 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.893602    2557 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 25 13:05:47.899575 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.893605    2557 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 25 13:05:47.899575 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.893607    2557 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 25 13:05:47.899575 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.893610    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 25 13:05:47.899575 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.893612    2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 25 13:05:47.899575 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.893615    2557 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 25 13:05:47.899575 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.893618    2557 feature_gate.go:328] unrecognized feature gate: Example
May 25 13:05:47.899575 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.893622    2557 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 25 13:05:47.899575 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.893625    2557 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 25 13:05:47.899575 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.893628    2557 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 25 13:05:47.899575 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.893631    2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 25 13:05:47.899575 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.893634    2557 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 25 13:05:47.899575 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.893637    2557 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 25 13:05:47.899575 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.893640    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 25 13:05:47.900041 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.893642    2557 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 25 13:05:47.900041 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.893646    2557 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 25 13:05:47.900041 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.893650    2557 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 25 13:05:47.900041 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.893653    2557 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 25 13:05:47.900041 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.893655    2557 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 25 13:05:47.900041 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.893660    2557 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 25 13:05:47.900041 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.893663    2557 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 25 13:05:47.900041 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.893665    2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 25 13:05:47.900041 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.893668    2557 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 25 13:05:47.900041 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.893671    2557 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 25 13:05:47.900041 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.893673    2557 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 25 13:05:47.900041 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.893676    2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 25 13:05:47.900041 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.893678    2557 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 25 13:05:47.900041 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.893681    2557 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 25 13:05:47.900041 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.893684    2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 25 13:05:47.900041 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.893686    2557 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 25 13:05:47.900041 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.893690    2557 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 25 13:05:47.900041 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.893693    2557 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 25 13:05:47.900041 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.893696    2557 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 25 13:05:47.900519 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.893699    2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 25 13:05:47.900519 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.893702    2557 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 25 13:05:47.900519 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.893704    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 25 13:05:47.900519 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.893707    2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 25 13:05:47.900519 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.893709    2557 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 25 13:05:47.900519 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.893712    2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 25 13:05:47.900519 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.893737    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 25 13:05:47.900519 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.893742    2557 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 25 13:05:47.900519 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.893745    2557 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 25 13:05:47.900519 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.893748    2557 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 25 13:05:47.900519 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.893750    2557 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 25 13:05:47.900519 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.893753    2557 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 25 13:05:47.900519 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.893756    2557 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 25 13:05:47.900519 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.893759    2557 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 25 13:05:47.900519 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.893762    2557 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 25 13:05:47.900519 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.893766    2557 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 25 13:05:47.900519 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.893769    2557 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 25 13:05:47.900519 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.893772    2557 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 25 13:05:47.900519 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.893776    2557 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 25 13:05:47.900519 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.893779    2557 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 25 13:05:47.901256 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.893781    2557 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 25 13:05:47.901256 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.893785    2557 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 25 13:05:47.901256 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.893788    2557 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 25 13:05:47.901256 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.893790    2557 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 25 13:05:47.901256 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.893793    2557 feature_gate.go:328] unrecognized feature gate: DualReplica
May 25 13:05:47.901256 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:47.895047    2557 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]}
May 25 13:05:47.902632 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:47.902607    2557 server.go:530] "Kubelet version" kubeletVersion="v1.33.11"
May 25 13:05:47.902632 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:47.902632    2557 server.go:532] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
May 25 13:05:47.902784 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.902702    2557 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 25 13:05:47.902784 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.902712    2557 feature_gate.go:328] unrecognized feature gate: DualReplica
May 25 13:05:47.902784 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.902717    2557 feature_gate.go:328] unrecognized feature gate: NewOLM
May 25 13:05:47.902784 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.902721    2557 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 25 13:05:47.902784 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.902725    2557 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 25 13:05:47.902784 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.902730    2557 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 25 13:05:47.902784 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.902735    2557 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 25 13:05:47.902784 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.902739    2557 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 25 13:05:47.902784 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.902744    2557 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 25 13:05:47.902784 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.902748    2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 25 13:05:47.902784 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.902753    2557 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 25 13:05:47.902784 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.902757    2557 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 25 13:05:47.902784 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.902761    2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 25 13:05:47.902784 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.902765    2557 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 25 13:05:47.902784 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.902770    2557 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 25 13:05:47.902784 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.902774    2557 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 25 13:05:47.902784 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.902779    2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 25 13:05:47.902784 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.902782    2557 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 25 13:05:47.902784 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.902787    2557 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 25 13:05:47.903634 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.902797    2557 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 25 13:05:47.903634 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.902802    2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 25 13:05:47.903634 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.902806    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 25 13:05:47.903634 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.902811    2557 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 25 13:05:47.903634 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.902815    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 25 13:05:47.903634 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.902819    2557 feature_gate.go:328] unrecognized feature gate: Example
May 25 13:05:47.903634 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.902823    2557 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 25 13:05:47.903634 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.902827    2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 25 13:05:47.903634 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.902831    2557 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 25 13:05:47.903634 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.902835    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 25 13:05:47.903634 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.902842    2557 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 25 13:05:47.903634 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.902848    2557 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 25 13:05:47.903634 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.902853    2557 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 25 13:05:47.903634 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.902857    2557 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 25 13:05:47.903634 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.902861    2557 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 25 13:05:47.903634 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.902865    2557 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 25 13:05:47.903634 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.902869    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 25 13:05:47.903634 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.902873    2557 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 25 13:05:47.903634 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.902878    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 25 13:05:47.904206 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.902882    2557 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 25 13:05:47.904206 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.902886    2557 feature_gate.go:328] unrecognized feature gate: Example2
May 25 13:05:47.904206 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.902890    2557 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 25 13:05:47.904206 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.902894    2557 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 25 13:05:47.904206 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.902898    2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 25 13:05:47.904206 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.902903    2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 25 13:05:47.904206 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.902907    2557 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 25 13:05:47.904206 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.902913    2557 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 25 13:05:47.904206 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.902920    2557 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 25 13:05:47.904206 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.902925    2557 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 25 13:05:47.904206 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.902929    2557 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 25 13:05:47.904206 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.902934    2557 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 25 13:05:47.904206 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.902938    2557 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 25 13:05:47.904206 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.902942    2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 25 13:05:47.904206 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.902949    2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 25 13:05:47.904206 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.902954    2557 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 25 13:05:47.904206 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.902958    2557 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 25 13:05:47.904206 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.902962    2557 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 25 13:05:47.904206 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.902966    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 25 13:05:47.904817 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.902971    2557 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 25 13:05:47.904817 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.902975    2557 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 25 13:05:47.904817 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.902979    2557 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 25 13:05:47.904817 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.902984    2557 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 25 13:05:47.904817 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.902987    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 25 13:05:47.904817 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.902993    2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 25 13:05:47.904817 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.902997    2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 25 13:05:47.904817 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.903001    2557 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 25 13:05:47.904817 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.903006    2557 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 25 13:05:47.904817 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.903010    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 25 13:05:47.904817 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.903014    2557 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 25 13:05:47.904817 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.903018    2557 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 25 13:05:47.904817 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.903022    2557 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 25 13:05:47.904817 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.903027    2557 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 25 13:05:47.904817 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.903031    2557 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 25 13:05:47.904817 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.903035    2557 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 25 13:05:47.904817 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.903039    2557 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 25 13:05:47.904817 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.903044    2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 25 13:05:47.904817 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.903048    2557 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 25 13:05:47.904817 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.903052    2557 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 25 13:05:47.905458 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.903056    2557 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 25 13:05:47.905458 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.903060    2557 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 25 13:05:47.905458 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.903064    2557 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 25 13:05:47.905458 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.903069    2557 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 25 13:05:47.905458 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.903073    2557 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 25 13:05:47.905458 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.903078    2557 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 25 13:05:47.905458 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.903083    2557 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 25 13:05:47.905458 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.903089    2557 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 25 13:05:47.905458 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.903094    2557 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 25 13:05:47.905458 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:47.903102    2557 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]}
May 25 13:05:47.905458 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.903284    2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 25 13:05:47.905458 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.903293    2557 feature_gate.go:328] unrecognized feature gate: NewOLM
May 25 13:05:47.905458 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.903298    2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 25 13:05:47.905458 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.903303    2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 25 13:05:47.905458 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.903307    2557 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 25 13:05:47.906007 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.903312    2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 25 13:05:47.906007 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.903317    2557 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 25 13:05:47.906007 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.903321    2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 25 13:05:47.906007 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.903326    2557 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 25 13:05:47.906007 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.903330    2557 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 25 13:05:47.906007 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.903335    2557 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 25 13:05:47.906007 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.903339    2557 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 25 13:05:47.906007 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.903359    2557 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 25 13:05:47.906007 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.903364    2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 25 13:05:47.906007 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.903368    2557 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 25 13:05:47.906007 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.903372    2557 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 25 13:05:47.906007 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.903377    2557 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 25 13:05:47.906007 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.903381    2557 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 25 13:05:47.906007 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.903384    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 25 13:05:47.906007 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.903388    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 25 13:05:47.906007 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.903393    2557 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 25 13:05:47.906007 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.903398    2557 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 25 13:05:47.906007 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.903402    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 25 13:05:47.906007 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.903407    2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 25 13:05:47.906007 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.903411    2557 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 25 13:05:47.906541 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.903415    2557 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 25 13:05:47.906541 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.903422    2557 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 25 13:05:47.906541 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.903429    2557 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 25 13:05:47.906541 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.903433    2557 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 25 13:05:47.906541 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.903439    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 25 13:05:47.906541 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.903444    2557 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 25 13:05:47.906541 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.903450    2557 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 25 13:05:47.906541 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.903455    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 25 13:05:47.906541 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.903459    2557 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 25 13:05:47.906541 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.903463    2557 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 25 13:05:47.906541 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.903467    2557 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 25 13:05:47.906541 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.903472    2557 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 25 13:05:47.906541 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.903476    2557 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 25 13:05:47.906541 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.903480    2557 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 25 13:05:47.906541 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.903485    2557 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 25 13:05:47.906541 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.903489    2557 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 25 13:05:47.906541 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.903493    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 25 13:05:47.906541 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.903497    2557 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 25 13:05:47.906541 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.903502    2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 25 13:05:47.906541 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.903506    2557 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 25 13:05:47.907333 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.903510    2557 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 25 13:05:47.907333 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.903514    2557 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 25 13:05:47.907333 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.903518    2557 feature_gate.go:328] unrecognized feature gate: DualReplica
May 25 13:05:47.907333 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.903523    2557 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 25 13:05:47.907333 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.903528    2557 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 25 13:05:47.907333 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.903532    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 25 13:05:47.907333 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.903536    2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 25 13:05:47.907333 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.903540    2557 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 25 13:05:47.907333 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.903544    2557 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 25 13:05:47.907333 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.903548    2557 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 25 13:05:47.907333 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.903552    2557 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 25 13:05:47.907333 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.903556    2557 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 25 13:05:47.907333 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.903560    2557 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 25 13:05:47.907333 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.903564    2557 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 25 13:05:47.907333 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.903569    2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 25 13:05:47.907333 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.903573    2557 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 25 13:05:47.907333 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.903577    2557 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 25 13:05:47.907333 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.903582    2557 feature_gate.go:328] unrecognized feature gate: Example
May 25 13:05:47.907333 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.903586    2557 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 25 13:05:47.907333 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.903591    2557 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 25 13:05:47.907983 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.903595    2557 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 25 13:05:47.907983 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.903599    2557 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 25 13:05:47.907983 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.903604    2557 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 25 13:05:47.907983 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.903609    2557 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 25 13:05:47.907983 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.903613    2557 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 25 13:05:47.907983 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.903617    2557 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 25 13:05:47.907983 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.903621    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 25 13:05:47.907983 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.903625    2557 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 25 13:05:47.907983 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.903629    2557 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 25 13:05:47.907983 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.903635    2557 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 25 13:05:47.907983 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.903641    2557 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 25 13:05:47.907983 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.903645    2557 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 25 13:05:47.907983 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.903649    2557 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 25 13:05:47.907983 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.903654    2557 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 25 13:05:47.907983 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.903658    2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 25 13:05:47.907983 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.903663    2557 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 25 13:05:47.907983 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.903667    2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 25 13:05:47.907983 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.903671    2557 feature_gate.go:328] unrecognized feature gate: Example2
May 25 13:05:47.907983 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.903675    2557 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 25 13:05:47.908594 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.903680    2557 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 25 13:05:47.908594 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:47.903684    2557 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 25 13:05:47.908594 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:47.903692    2557 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]}
May 25 13:05:47.908594 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:47.904566    2557 server.go:962] "Client rotation is on, will bootstrap in background"
May 25 13:05:47.908594 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:47.908365    2557 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
May 25 13:05:47.909473 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:47.909461    2557 server.go:1019] "Starting client certificate rotation"
May 25 13:05:47.909591 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:47.909570    2557 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
May 25 13:05:47.909634 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:47.909613    2557 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
May 25 13:05:47.937979 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:47.937943    2557 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
May 25 13:05:47.939297 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:47.939267    2557 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
May 25 13:05:47.959492 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:47.959465    2557 log.go:25] "Validated CRI v1 runtime API"
May 25 13:05:47.967127 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:47.967100    2557 log.go:25] "Validated CRI v1 image API"
May 25 13:05:47.967630 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:47.967614    2557 reflector.go:430] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
May 25 13:05:47.969282 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:47.969225    2557 server.go:1452] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
May 25 13:05:47.972359 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:47.972322    2557 fs.go:135] Filesystem UUIDs: map[604ab27d-a624-4b4d-af40-c88766c966a3:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 bb069edf-b4c5-4797-8ab8-801aead2f846:/dev/nvme0n1p3]
May 25 13:05:47.972436 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:47.972358    2557 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
May 25 13:05:47.976876 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:47.976752    2557 manager.go:217] Machine: {Timestamp:2026-05-25 13:05:47.976158483 +0000 UTC m=+0.439174647 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3103973 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2286acef130b4c5682832e8c3bd174 SystemUUID:ec2286ac-ef13-0b4c-5682-832e8c3bd174 BootID:6ea9f554-8612-48d5-a8fa-f43bc0deda0b Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6103040 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:98:e6:96:5d:0f Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:98:e6:96:5d:0f Speed:0 Mtu:9001} {Name:ovs-system MacAddress:a6:0a:d5:c4:99:30 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
May 25 13:05:47.977619 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:47.977605    2557 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
May 25 13:05:47.977714 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:47.977702    2557 manager.go:233] Version: {KernelVersion:5.14.0-570.114.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260512-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
May 25 13:05:47.981418 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:47.981384    2557 container_manager_linux.go:270] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
May 25 13:05:47.981567 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:47.981420    2557 container_manager_linux.go:275] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-143-235.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
May 25 13:05:47.981610 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:47.981577    2557 topology_manager.go:138] "Creating topology manager with none policy"
May 25 13:05:47.981610 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:47.981587    2557 container_manager_linux.go:306] "Creating device plugin manager"
May 25 13:05:47.981610 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:47.981595    2557 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
May 25 13:05:47.982396 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:47.982385    2557 server.go:72] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
May 25 13:05:47.983888 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:47.983878    2557 state_mem.go:36] "Initialized new in-memory state store"
May 25 13:05:47.984002 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:47.983993    2557 server.go:1267] "Using root directory" path="/var/lib/kubelet"
May 25 13:05:47.987380 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:47.987341    2557 kubelet.go:491] "Attempting to sync node with API server"
May 25 13:05:47.987380 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:47.987378    2557 kubelet.go:386] "Adding static pod path" path="/etc/kubernetes/manifests"
May 25 13:05:47.987539 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:47.987391    2557 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
May 25 13:05:47.987539 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:47.987401    2557 kubelet.go:397] "Adding apiserver pod source"
May 25 13:05:47.987539 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:47.987411    2557 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
May 25 13:05:47.988647 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:47.988633    2557 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
May 25 13:05:47.988703 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:47.988654    2557 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
May 25 13:05:47.992267 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:47.992248    2557 kuberuntime_manager.go:279] "Container runtime initialized" containerRuntime="cri-o" version="1.33.11-2.rhaos4.20.gitb2a8320.el9" apiVersion="v1"
May 25 13:05:47.993306 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:47.993286    2557 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-jd4kq"
May 25 13:05:47.994617 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:47.994599    2557 kubelet.go:953] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
May 25 13:05:47.995533 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:47.995519    2557 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
May 25 13:05:47.995603 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:47.995540    2557 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
May 25 13:05:47.995603 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:47.995548    2557 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
May 25 13:05:47.995603 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:47.995557    2557 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
May 25 13:05:47.995603 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:47.995568    2557 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
May 25 13:05:47.995603 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:47.995577    2557 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/secret"
May 25 13:05:47.995603 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:47.995585    2557 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
May 25 13:05:47.995603 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:47.995594    2557 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
May 25 13:05:47.995603 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:47.995606    2557 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/fc"
May 25 13:05:47.995834 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:47.995615    2557 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
May 25 13:05:47.995834 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:47.995635    2557 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/projected"
May 25 13:05:47.995943 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:47.995932    2557 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
May 25 13:05:47.997123 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:47.997113    2557 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/csi"
May 25 13:05:47.997185 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:47.997135    2557 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/image"
May 25 13:05:47.999403 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:47.999382    2557 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-jd4kq"
May 25 13:05:48.000579 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:48.000562    2557 watchdog_linux.go:99] "Systemd watchdog is not enabled"
May 25 13:05:48.000657 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:48.000609    2557 server.go:1295] "Started kubelet"
May 25 13:05:48.000701 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:48.000676    2557 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
May 25 13:05:48.003783 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:48.003717    2557 ratelimit.go:55] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
May 25 13:05:48.003868 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:48.003816    2557 server_v1.go:47] "podresources" method="list" useActivePods=true
May 25 13:05:48.004740 ip-10-0-143-235 systemd[1]: Started Kubernetes Kubelet.
May 25 13:05:48.005373 ip-10-0-143-235 kubenswrapper[2557]: E0525 13:05:48.005213    2557 reflector.go:200] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-143-235.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
May 25 13:05:48.005373 ip-10-0-143-235 kubenswrapper[2557]: E0525 13:05:48.005341    2557 reflector.go:200] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
May 25 13:05:48.005488 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:48.005443    2557 csi_plugin.go:988] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-235.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
May 25 13:05:48.006332 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:48.006312    2557 server.go:255] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
May 25 13:05:48.007006 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:48.006981    2557 server.go:317] "Adding debug handlers to kubelet server"
May 25 13:05:48.012239 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:48.012211    2557 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
May 25 13:05:48.012239 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:48.012224    2557 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
May 25 13:05:48.012406 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:48.012278    2557 volume_manager.go:295] "The desired_state_of_world populator starts"
May 25 13:05:48.012406 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:48.012286    2557 volume_manager.go:297] "Starting Kubelet Volume Manager"
May 25 13:05:48.012519 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:48.012450    2557 desired_state_of_world_populator.go:150] "Desired state populator starts to run"
May 25 13:05:48.012519 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:48.012495    2557 reconstruct.go:97] "Volume reconstruction finished"
May 25 13:05:48.012519 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:48.012505    2557 reconciler.go:26] "Reconciler: start to sync state"
May 25 13:05:48.013052 ip-10-0-143-235 kubenswrapper[2557]: E0525 13:05:48.013024    2557 kubelet.go:1618] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
May 25 13:05:48.013165 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:48.013128    2557 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
May 25 13:05:48.013165 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:48.013143    2557 factory.go:55] Registering systemd factory
May 25 13:05:48.013165 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:48.013152    2557 factory.go:223] Registration of the systemd container factory successfully
May 25 13:05:48.013389 ip-10-0-143-235 kubenswrapper[2557]: E0525 13:05:48.013367    2557 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-143-235.ec2.internal\" not found"
May 25 13:05:48.013389 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:48.013380    2557 factory.go:153] Registering CRI-O factory
May 25 13:05:48.013475 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:48.013401    2557 factory.go:223] Registration of the crio container factory successfully
May 25 13:05:48.013475 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:48.013416    2557 factory.go:103] Registering Raw factory
May 25 13:05:48.013475 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:48.013429    2557 manager.go:1196] Started watching for new ooms in manager
May 25 13:05:48.013756 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:48.013745    2557 manager.go:319] Starting recovery of all containers
May 25 13:05:48.016113 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:48.016093    2557 reflector.go:430] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
May 25 13:05:48.019076 ip-10-0-143-235 kubenswrapper[2557]: E0525 13:05:48.018909    2557 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-143-235.ec2.internal\" not found" node="ip-10-0-143-235.ec2.internal"
May 25 13:05:48.023543 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:48.023526    2557 manager.go:324] Recovery completed
May 25 13:05:48.027963 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:48.027950    2557 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 25 13:05:48.029859 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:48.029843    2557 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-143-235.ec2.internal" event="NodeHasSufficientMemory"
May 25 13:05:48.029917 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:48.029873    2557 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-143-235.ec2.internal" event="NodeHasNoDiskPressure"
May 25 13:05:48.029917 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:48.029884    2557 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-143-235.ec2.internal" event="NodeHasSufficientPID"
May 25 13:05:48.030444 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:48.030429    2557 cpu_manager.go:222] "Starting CPU manager" policy="none"
May 25 13:05:48.030444 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:48.030440    2557 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
May 25 13:05:48.030556 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:48.030466    2557 state_mem.go:36] "Initialized new in-memory state store"
May 25 13:05:48.032754 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:48.032742    2557 policy_none.go:49] "None policy: Start"
May 25 13:05:48.032812 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:48.032759    2557 memory_manager.go:186] "Starting memorymanager" policy="None"
May 25 13:05:48.032812 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:48.032769    2557 state_mem.go:35] "Initializing new in-memory state store"
May 25 13:05:48.080109 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:48.080087    2557 manager.go:341] "Starting Device Plugin manager"
May 25 13:05:48.081301 ip-10-0-143-235 kubenswrapper[2557]: E0525 13:05:48.080125    2557 manager.go:517] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
May 25 13:05:48.081301 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:48.080136    2557 server.go:85] "Starting device plugin registration server"
May 25 13:05:48.081301 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:48.080467    2557 eviction_manager.go:189] "Eviction manager: starting control loop"
May 25 13:05:48.081301 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:48.080483    2557 container_log_manager.go:189] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
May 25 13:05:48.081301 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:48.080615    2557 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
May 25 13:05:48.081301 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:48.080697    2557 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
May 25 13:05:48.081301 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:48.080705    2557 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
May 25 13:05:48.081301 ip-10-0-143-235 kubenswrapper[2557]: E0525 13:05:48.081261    2557 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
May 25 13:05:48.081301 ip-10-0-143-235 kubenswrapper[2557]: E0525 13:05:48.081302    2557 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-143-235.ec2.internal\" not found"
May 25 13:05:48.143826 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:48.143739    2557 kubelet_network_linux.go:49] "Initialized iptables rules." protocol="IPv4"
May 25 13:05:48.144936 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:48.144918    2557 kubelet_network_linux.go:49] "Initialized iptables rules." protocol="IPv6"
May 25 13:05:48.145003 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:48.144943    2557 status_manager.go:230] "Starting to sync pod status with apiserver"
May 25 13:05:48.145003 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:48.144962    2557 watchdog_linux.go:127] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
May 25 13:05:48.145003 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:48.144972    2557 kubelet.go:2451] "Starting kubelet main sync loop"
May 25 13:05:48.145119 ip-10-0-143-235 kubenswrapper[2557]: E0525 13:05:48.145019    2557 kubelet.go:2475] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
May 25 13:05:48.148333 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:48.148304    2557 reflector.go:430] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
May 25 13:05:48.181281 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:48.181254    2557 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 25 13:05:48.186730 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:48.186711    2557 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-143-235.ec2.internal" event="NodeHasSufficientMemory"
May 25 13:05:48.186852 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:48.186743    2557 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-143-235.ec2.internal" event="NodeHasNoDiskPressure"
May 25 13:05:48.186852 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:48.186754    2557 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-143-235.ec2.internal" event="NodeHasSufficientPID"
May 25 13:05:48.186852 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:48.186779    2557 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-143-235.ec2.internal"
May 25 13:05:48.196390 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:48.196366    2557 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-143-235.ec2.internal"
May 25 13:05:48.196390 ip-10-0-143-235 kubenswrapper[2557]: E0525 13:05:48.196392    2557 kubelet_node_status.go:597] "Error updating node status, will retry" err="error getting node \"ip-10-0-143-235.ec2.internal\": node \"ip-10-0-143-235.ec2.internal\" not found"
May 25 13:05:48.210124 ip-10-0-143-235 kubenswrapper[2557]: E0525 13:05:48.210096    2557 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-143-235.ec2.internal\" not found"
May 25 13:05:48.245766 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:48.245719    2557 kubelet.go:2537] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-235.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-143-235.ec2.internal"]
May 25 13:05:48.245848 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:48.245817    2557 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 25 13:05:48.246834 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:48.246816    2557 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-143-235.ec2.internal" event="NodeHasSufficientMemory"
May 25 13:05:48.246971 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:48.246847    2557 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-143-235.ec2.internal" event="NodeHasNoDiskPressure"
May 25 13:05:48.246971 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:48.246857    2557 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-143-235.ec2.internal" event="NodeHasSufficientPID"
May 25 13:05:48.248240 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:48.248228    2557 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 25 13:05:48.248414 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:48.248400    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-235.ec2.internal"
May 25 13:05:48.248472 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:48.248430    2557 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 25 13:05:48.249075 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:48.249059    2557 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-143-235.ec2.internal" event="NodeHasSufficientMemory"
May 25 13:05:48.249126 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:48.249087    2557 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-143-235.ec2.internal" event="NodeHasNoDiskPressure"
May 25 13:05:48.249126 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:48.249097    2557 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-143-235.ec2.internal" event="NodeHasSufficientPID"
May 25 13:05:48.249199 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:48.249063    2557 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-143-235.ec2.internal" event="NodeHasSufficientMemory"
May 25 13:05:48.249199 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:48.249163    2557 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-143-235.ec2.internal" event="NodeHasNoDiskPressure"
May 25 13:05:48.249199 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:48.249176    2557 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-143-235.ec2.internal" event="NodeHasSufficientPID"
May 25 13:05:48.250268 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:48.250255    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-235.ec2.internal"
May 25 13:05:48.250312 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:48.250279    2557 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 25 13:05:48.251003 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:48.250987    2557 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-143-235.ec2.internal" event="NodeHasSufficientMemory"
May 25 13:05:48.251066 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:48.251014    2557 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-143-235.ec2.internal" event="NodeHasNoDiskPressure"
May 25 13:05:48.251066 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:48.251026    2557 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-143-235.ec2.internal" event="NodeHasSufficientPID"
May 25 13:05:48.264960 ip-10-0-143-235 kubenswrapper[2557]: E0525 13:05:48.264936    2557 kubelet.go:3336] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-235.ec2.internal\" not found" node="ip-10-0-143-235.ec2.internal"
May 25 13:05:48.269043 ip-10-0-143-235 kubenswrapper[2557]: E0525 13:05:48.269025    2557 kubelet.go:3336] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-235.ec2.internal\" not found" node="ip-10-0-143-235.ec2.internal"
May 25 13:05:48.310642 ip-10-0-143-235 kubenswrapper[2557]: E0525 13:05:48.310600    2557 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-143-235.ec2.internal\" not found"
May 25 13:05:48.315708 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:48.315689    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f10bc5abc6a23ce724748b3d53d4900b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-235.ec2.internal\" (UID: \"f10bc5abc6a23ce724748b3d53d4900b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-235.ec2.internal"
May 25 13:05:48.315764 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:48.315718    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f10bc5abc6a23ce724748b3d53d4900b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-235.ec2.internal\" (UID: \"f10bc5abc6a23ce724748b3d53d4900b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-235.ec2.internal"
May 25 13:05:48.315764 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:48.315741    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/be380709ced6c0e0087779b84acc1278-config\") pod \"kube-apiserver-proxy-ip-10-0-143-235.ec2.internal\" (UID: \"be380709ced6c0e0087779b84acc1278\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-235.ec2.internal"
May 25 13:05:48.411701 ip-10-0-143-235 kubenswrapper[2557]: E0525 13:05:48.411633    2557 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-143-235.ec2.internal\" not found"
May 25 13:05:48.415880 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:48.415847    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f10bc5abc6a23ce724748b3d53d4900b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-235.ec2.internal\" (UID: \"f10bc5abc6a23ce724748b3d53d4900b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-235.ec2.internal"
May 25 13:05:48.415946 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:48.415892    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f10bc5abc6a23ce724748b3d53d4900b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-235.ec2.internal\" (UID: \"f10bc5abc6a23ce724748b3d53d4900b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-235.ec2.internal"
May 25 13:05:48.415946 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:48.415920    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/be380709ced6c0e0087779b84acc1278-config\") pod \"kube-apiserver-proxy-ip-10-0-143-235.ec2.internal\" (UID: \"be380709ced6c0e0087779b84acc1278\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-235.ec2.internal"
May 25 13:05:48.416019 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:48.415953    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f10bc5abc6a23ce724748b3d53d4900b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-235.ec2.internal\" (UID: \"f10bc5abc6a23ce724748b3d53d4900b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-235.ec2.internal"
May 25 13:05:48.416019 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:48.415974    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f10bc5abc6a23ce724748b3d53d4900b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-235.ec2.internal\" (UID: \"f10bc5abc6a23ce724748b3d53d4900b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-235.ec2.internal"
May 25 13:05:48.416089 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:48.416016    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/be380709ced6c0e0087779b84acc1278-config\") pod \"kube-apiserver-proxy-ip-10-0-143-235.ec2.internal\" (UID: \"be380709ced6c0e0087779b84acc1278\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-235.ec2.internal"
May 25 13:05:48.512224 ip-10-0-143-235 kubenswrapper[2557]: E0525 13:05:48.512182    2557 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-143-235.ec2.internal\" not found"
May 25 13:05:48.567558 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:48.567525    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-235.ec2.internal"
May 25 13:05:48.571218 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:48.571196    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-235.ec2.internal"
May 25 13:05:48.612691 ip-10-0-143-235 kubenswrapper[2557]: E0525 13:05:48.612663    2557 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-143-235.ec2.internal\" not found"
May 25 13:05:48.713098 ip-10-0-143-235 kubenswrapper[2557]: E0525 13:05:48.713017    2557 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-143-235.ec2.internal\" not found"
May 25 13:05:48.813424 ip-10-0-143-235 kubenswrapper[2557]: E0525 13:05:48.813389    2557 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-143-235.ec2.internal\" not found"
May 25 13:05:48.900334 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:48.900302    2557 reflector.go:430] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
May 25 13:05:48.909355 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:48.909321    2557 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
May 25 13:05:48.909484 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:48.909460    2557 reflector.go:556] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
May 25 13:05:48.909520 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:48.909462    2557 reflector.go:556] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
May 25 13:05:48.909520 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:48.909459    2557 reflector.go:556] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
May 25 13:05:48.913488 ip-10-0-143-235 kubenswrapper[2557]: E0525 13:05:48.913469    2557 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-143-235.ec2.internal\" not found"
May 25 13:05:49.004184 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:49.004103    2557 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-05-24 13:00:47 +0000 UTC" deadline="2027-11-08 23:41:05.515379067 +0000 UTC"
May 25 13:05:49.004184 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:49.004141    2557 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12778h35m16.511242052s"
May 25 13:05:49.012557 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:49.012529    2557 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
May 25 13:05:49.013611 ip-10-0-143-235 kubenswrapper[2557]: E0525 13:05:49.013590    2557 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-143-235.ec2.internal\" not found"
May 25 13:05:49.031110 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:49.031085    2557 reflector.go:430] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
May 25 13:05:49.051000 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:49.050968    2557 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-gr2s5"
May 25 13:05:49.054835 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:49.054806    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf10bc5abc6a23ce724748b3d53d4900b.slice/crio-1b10831dbc95b395ad58f26004d77a5685430457a7d81c1cc497bf9a8d98f726 WatchSource:0}: Error finding container 1b10831dbc95b395ad58f26004d77a5685430457a7d81c1cc497bf9a8d98f726: Status 404 returned error can't find the container with id 1b10831dbc95b395ad58f26004d77a5685430457a7d81c1cc497bf9a8d98f726
May 25 13:05:49.055055 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:49.055027    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe380709ced6c0e0087779b84acc1278.slice/crio-b639682ab5db087702babaed4c38929bbefc38a6b94e44d80ef52eb57f8b3ee9 WatchSource:0}: Error finding container b639682ab5db087702babaed4c38929bbefc38a6b94e44d80ef52eb57f8b3ee9: Status 404 returned error can't find the container with id b639682ab5db087702babaed4c38929bbefc38a6b94e44d80ef52eb57f8b3ee9
May 25 13:05:49.059097 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:49.059081    2557 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-gr2s5"
May 25 13:05:49.059685 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:49.059670    2557 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 25 13:05:49.114717 ip-10-0-143-235 kubenswrapper[2557]: E0525 13:05:49.114679    2557 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-143-235.ec2.internal\" not found"
May 25 13:05:49.148014 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:49.147968    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-235.ec2.internal" event={"ID":"f10bc5abc6a23ce724748b3d53d4900b","Type":"ContainerStarted","Data":"1b10831dbc95b395ad58f26004d77a5685430457a7d81c1cc497bf9a8d98f726"}
May 25 13:05:49.148976 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:49.148955    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-235.ec2.internal" event={"ID":"be380709ced6c0e0087779b84acc1278","Type":"ContainerStarted","Data":"b639682ab5db087702babaed4c38929bbefc38a6b94e44d80ef52eb57f8b3ee9"}
May 25 13:05:49.215316 ip-10-0-143-235 kubenswrapper[2557]: E0525 13:05:49.215280    2557 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-143-235.ec2.internal\" not found"
May 25 13:05:49.284513 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:49.284431    2557 reflector.go:430] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
May 25 13:05:49.313005 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:49.312973    2557 kubelet.go:3340] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-235.ec2.internal"
May 25 13:05:49.325206 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:49.325179    2557 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
May 25 13:05:49.326050 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:49.326036    2557 kubelet.go:3340] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-235.ec2.internal"
May 25 13:05:49.332232 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:49.332206    2557 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
May 25 13:05:49.847029 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:49.846991    2557 reflector.go:430] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
May 25 13:05:49.988912 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:49.988822    2557 apiserver.go:52] "Watching apiserver"
May 25 13:05:49.997373 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:49.997332    2557 reflector.go:430] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
May 25 13:05:49.997685 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:49.997655    2557 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-network-operator/iptables-alerter-bwz4h","kube-system/kube-apiserver-proxy-ip-10-0-143-235.ec2.internal","openshift-cluster-node-tuning-operator/tuned-lcmng","openshift-image-registry/node-ca-v9wvs","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-235.ec2.internal","openshift-multus/multus-additional-cni-plugins-lphh6","openshift-multus/multus-scw67","openshift-multus/network-metrics-daemon-5lm4j","openshift-ovn-kubernetes/ovnkube-node-vqnvc","kube-system/konnectivity-agent-vwnkg","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jc4n4","openshift-network-diagnostics/network-check-target-96mr4"]
May 25 13:05:49.999084 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:49.999066    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-bwz4h"
May 25 13:05:50.001310 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.001289    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-lcmng"
May 25 13:05:50.001772 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.001748    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
May 25 13:05:50.001929 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.001827    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
May 25 13:05:50.001929 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.001883    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-kfwjb\""
May 25 13:05:50.002069 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.002050    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
May 25 13:05:50.002631 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.002615    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-v9wvs"
May 25 13:05:50.003687 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.003668    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-tklm8\""
May 25 13:05:50.003943 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.003802    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
May 25 13:05:50.003943 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.003876    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
May 25 13:05:50.003943 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.003894    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-lphh6"
May 25 13:05:50.005039 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.004988    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
May 25 13:05:50.005127 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.005080    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-scw67"
May 25 13:05:50.005219 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.005079    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
May 25 13:05:50.005470 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.005451    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
May 25 13:05:50.005568 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.005507    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-9x8rn\""
May 25 13:05:50.006303 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.006287    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
May 25 13:05:50.006409 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.006395    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
May 25 13:05:50.006580 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.006528    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5lm4j"
May 25 13:05:50.006667 ip-10-0-143-235 kubenswrapper[2557]: E0525 13:05:50.006598    2557 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5lm4j" podUID="8c2f1987-8965-449c-b426-b46515332392"
May 25 13:05:50.006754 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.006736    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
May 25 13:05:50.006754 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.006748    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-vqnvc"
May 25 13:05:50.006910 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.006822    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-cdfqp\""
May 25 13:05:50.006910 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.006879    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
May 25 13:05:50.007377 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.006940    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
May 25 13:05:50.007713 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.007697    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
May 25 13:05:50.007803 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.007743    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-ljhgm\""
May 25 13:05:50.007871 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.007833    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-vwnkg"
May 25 13:05:50.009140 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.009122    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jc4n4"
May 25 13:05:50.009238 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.009224    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
May 25 13:05:50.009333 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.009225    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
May 25 13:05:50.009569 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.009550    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
May 25 13:05:50.009658 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.009584    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
May 25 13:05:50.010548 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.010513    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-96mr4"
May 25 13:05:50.010648 ip-10-0-143-235 kubenswrapper[2557]: E0525 13:05:50.010586    2557 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-96mr4" podUID="a8d6074e-8880-4fd2-b612-de33be9d873e"
May 25 13:05:50.010731 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.010648    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
May 25 13:05:50.011451 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.011434    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-bsq9r\""
May 25 13:05:50.012107 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.011905    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
May 25 13:05:50.012107 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.011932    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
May 25 13:05:50.012107 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.011947    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-pc95k\""
May 25 13:05:50.012107 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.011966    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
May 25 13:05:50.012331 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.012188    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
May 25 13:05:50.012331 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.012260    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
May 25 13:05:50.012903 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.012834    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
May 25 13:05:50.012903 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.012894    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-k242x\""
May 25 13:05:50.015464 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.014881    2557 desired_state_of_world_populator.go:158] "Finished populating initial desired state of world"
May 25 13:05:50.024932 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.024910    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/904a1760-12f0-49b4-a389-c71b39a99619-etc-kubernetes\") pod \"tuned-lcmng\" (UID: \"904a1760-12f0-49b4-a389-c71b39a99619\") " pod="openshift-cluster-node-tuning-operator/tuned-lcmng"
May 25 13:05:50.025040 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.024945    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d21bdf8b-d0ab-48f2-8736-54e92947dd53-host-run-netns\") pod \"ovnkube-node-vqnvc\" (UID: \"d21bdf8b-d0ab-48f2-8736-54e92947dd53\") " pod="openshift-ovn-kubernetes/ovnkube-node-vqnvc"
May 25 13:05:50.025040 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.024973    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96fp8\" (UniqueName: \"kubernetes.io/projected/d21bdf8b-d0ab-48f2-8736-54e92947dd53-kube-api-access-96fp8\") pod \"ovnkube-node-vqnvc\" (UID: \"d21bdf8b-d0ab-48f2-8736-54e92947dd53\") " pod="openshift-ovn-kubernetes/ovnkube-node-vqnvc"
May 25 13:05:50.025040 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.024999    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/904a1760-12f0-49b4-a389-c71b39a99619-etc-sysconfig\") pod \"tuned-lcmng\" (UID: \"904a1760-12f0-49b4-a389-c71b39a99619\") " pod="openshift-cluster-node-tuning-operator/tuned-lcmng"
May 25 13:05:50.025040 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.025023    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvzsd\" (UniqueName: \"kubernetes.io/projected/904a1760-12f0-49b4-a389-c71b39a99619-kube-api-access-kvzsd\") pod \"tuned-lcmng\" (UID: \"904a1760-12f0-49b4-a389-c71b39a99619\") " pod="openshift-cluster-node-tuning-operator/tuned-lcmng"
May 25 13:05:50.025239 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.025049    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/87942b76-e2a7-407e-9a99-2ea92f737157-multus-conf-dir\") pod \"multus-scw67\" (UID: \"87942b76-e2a7-407e-9a99-2ea92f737157\") " pod="openshift-multus/multus-scw67"
May 25 13:05:50.025239 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.025072    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d21bdf8b-d0ab-48f2-8736-54e92947dd53-ovnkube-script-lib\") pod \"ovnkube-node-vqnvc\" (UID: \"d21bdf8b-d0ab-48f2-8736-54e92947dd53\") " pod="openshift-ovn-kubernetes/ovnkube-node-vqnvc"
May 25 13:05:50.025239 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.025095    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/904a1760-12f0-49b4-a389-c71b39a99619-host\") pod \"tuned-lcmng\" (UID: \"904a1760-12f0-49b4-a389-c71b39a99619\") " pod="openshift-cluster-node-tuning-operator/tuned-lcmng"
May 25 13:05:50.025239 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.025122    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/87942b76-e2a7-407e-9a99-2ea92f737157-host-run-netns\") pod \"multus-scw67\" (UID: \"87942b76-e2a7-407e-9a99-2ea92f737157\") " pod="openshift-multus/multus-scw67"
May 25 13:05:50.025239 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.025153    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d21bdf8b-d0ab-48f2-8736-54e92947dd53-host-kubelet\") pod \"ovnkube-node-vqnvc\" (UID: \"d21bdf8b-d0ab-48f2-8736-54e92947dd53\") " pod="openshift-ovn-kubernetes/ovnkube-node-vqnvc"
May 25 13:05:50.025239 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.025179    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/904a1760-12f0-49b4-a389-c71b39a99619-sys\") pod \"tuned-lcmng\" (UID: \"904a1760-12f0-49b4-a389-c71b39a99619\") " pod="openshift-cluster-node-tuning-operator/tuned-lcmng"
May 25 13:05:50.025239 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.025203    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/87942b76-e2a7-407e-9a99-2ea92f737157-system-cni-dir\") pod \"multus-scw67\" (UID: \"87942b76-e2a7-407e-9a99-2ea92f737157\") " pod="openshift-multus/multus-scw67"
May 25 13:05:50.025239 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.025229    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/87942b76-e2a7-407e-9a99-2ea92f737157-host-var-lib-cni-multus\") pod \"multus-scw67\" (UID: \"87942b76-e2a7-407e-9a99-2ea92f737157\") " pod="openshift-multus/multus-scw67"
May 25 13:05:50.025622 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.025255    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xx67l\" (UniqueName: \"kubernetes.io/projected/87942b76-e2a7-407e-9a99-2ea92f737157-kube-api-access-xx67l\") pod \"multus-scw67\" (UID: \"87942b76-e2a7-407e-9a99-2ea92f737157\") " pod="openshift-multus/multus-scw67"
May 25 13:05:50.025622 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.025283    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d21bdf8b-d0ab-48f2-8736-54e92947dd53-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vqnvc\" (UID: \"d21bdf8b-d0ab-48f2-8736-54e92947dd53\") " pod="openshift-ovn-kubernetes/ovnkube-node-vqnvc"
May 25 13:05:50.025622 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.025309    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/904a1760-12f0-49b4-a389-c71b39a99619-etc-systemd\") pod \"tuned-lcmng\" (UID: \"904a1760-12f0-49b4-a389-c71b39a99619\") " pod="openshift-cluster-node-tuning-operator/tuned-lcmng"
May 25 13:05:50.025622 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.025334    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/87942b76-e2a7-407e-9a99-2ea92f737157-multus-daemon-config\") pod \"multus-scw67\" (UID: \"87942b76-e2a7-407e-9a99-2ea92f737157\") " pod="openshift-multus/multus-scw67"
May 25 13:05:50.025622 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.025377    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1dcac962-5867-4644-b3b3-ce62b3216b79-system-cni-dir\") pod \"multus-additional-cni-plugins-lphh6\" (UID: \"1dcac962-5867-4644-b3b3-ce62b3216b79\") " pod="openshift-multus/multus-additional-cni-plugins-lphh6"
May 25 13:05:50.025622 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.025417    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9d77\" (UniqueName: \"kubernetes.io/projected/1dcac962-5867-4644-b3b3-ce62b3216b79-kube-api-access-w9d77\") pod \"multus-additional-cni-plugins-lphh6\" (UID: \"1dcac962-5867-4644-b3b3-ce62b3216b79\") " pod="openshift-multus/multus-additional-cni-plugins-lphh6"
May 25 13:05:50.025622 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.025460    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d21bdf8b-d0ab-48f2-8736-54e92947dd53-run-ovn\") pod \"ovnkube-node-vqnvc\" (UID: \"d21bdf8b-d0ab-48f2-8736-54e92947dd53\") " pod="openshift-ovn-kubernetes/ovnkube-node-vqnvc"
May 25 13:05:50.025622 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.025488    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d21bdf8b-d0ab-48f2-8736-54e92947dd53-host-cni-netd\") pod \"ovnkube-node-vqnvc\" (UID: \"d21bdf8b-d0ab-48f2-8736-54e92947dd53\") " pod="openshift-ovn-kubernetes/ovnkube-node-vqnvc"
May 25 13:05:50.025622 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.025515    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/904a1760-12f0-49b4-a389-c71b39a99619-run\") pod \"tuned-lcmng\" (UID: \"904a1760-12f0-49b4-a389-c71b39a99619\") " pod="openshift-cluster-node-tuning-operator/tuned-lcmng"
May 25 13:05:50.025622 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.025538    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/87942b76-e2a7-407e-9a99-2ea92f737157-os-release\") pod \"multus-scw67\" (UID: \"87942b76-e2a7-407e-9a99-2ea92f737157\") " pod="openshift-multus/multus-scw67"
May 25 13:05:50.025622 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.025568    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9d1a5d53-d761-478d-a85e-0974f65121e3-serviceca\") pod \"node-ca-v9wvs\" (UID: \"9d1a5d53-d761-478d-a85e-0974f65121e3\") " pod="openshift-image-registry/node-ca-v9wvs"
May 25 13:05:50.025622 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.025593    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c2f1987-8965-449c-b426-b46515332392-metrics-certs\") pod \"network-metrics-daemon-5lm4j\" (UID: \"8c2f1987-8965-449c-b426-b46515332392\") " pod="openshift-multus/network-metrics-daemon-5lm4j"
May 25 13:05:50.025622 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.025615    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d21bdf8b-d0ab-48f2-8736-54e92947dd53-run-openvswitch\") pod \"ovnkube-node-vqnvc\" (UID: \"d21bdf8b-d0ab-48f2-8736-54e92947dd53\") " pod="openshift-ovn-kubernetes/ovnkube-node-vqnvc"
May 25 13:05:50.026157 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.025641    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/71fb8f1c-14f2-441f-bbbf-92696b6559af-host-slash\") pod \"iptables-alerter-bwz4h\" (UID: \"71fb8f1c-14f2-441f-bbbf-92696b6559af\") " pod="openshift-network-operator/iptables-alerter-bwz4h"
May 25 13:05:50.026157 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.025663    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/87942b76-e2a7-407e-9a99-2ea92f737157-host-var-lib-cni-bin\") pod \"multus-scw67\" (UID: \"87942b76-e2a7-407e-9a99-2ea92f737157\") " pod="openshift-multus/multus-scw67"
May 25 13:05:50.026157 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.025686    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bk5mv\" (UniqueName: \"kubernetes.io/projected/9d1a5d53-d761-478d-a85e-0974f65121e3-kube-api-access-bk5mv\") pod \"node-ca-v9wvs\" (UID: \"9d1a5d53-d761-478d-a85e-0974f65121e3\") " pod="openshift-image-registry/node-ca-v9wvs"
May 25 13:05:50.026157 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.025729    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1dcac962-5867-4644-b3b3-ce62b3216b79-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-lphh6\" (UID: \"1dcac962-5867-4644-b3b3-ce62b3216b79\") " pod="openshift-multus/multus-additional-cni-plugins-lphh6"
May 25 13:05:50.026157 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.025784    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d21bdf8b-d0ab-48f2-8736-54e92947dd53-host-slash\") pod \"ovnkube-node-vqnvc\" (UID: \"d21bdf8b-d0ab-48f2-8736-54e92947dd53\") " pod="openshift-ovn-kubernetes/ovnkube-node-vqnvc"
May 25 13:05:50.026157 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.025823    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pb9vv\" (UniqueName: \"kubernetes.io/projected/b3857de1-9f56-4063-b132-410d217c9f8b-kube-api-access-pb9vv\") pod \"aws-ebs-csi-driver-node-jc4n4\" (UID: \"b3857de1-9f56-4063-b132-410d217c9f8b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jc4n4"
May 25 13:05:50.026157 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.025850    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4f7rk\" (UniqueName: \"kubernetes.io/projected/a8d6074e-8880-4fd2-b612-de33be9d873e-kube-api-access-4f7rk\") pod \"network-check-target-96mr4\" (UID: \"a8d6074e-8880-4fd2-b612-de33be9d873e\") " pod="openshift-network-diagnostics/network-check-target-96mr4"
May 25 13:05:50.026157 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.025874    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/1dcac962-5867-4644-b3b3-ce62b3216b79-os-release\") pod \"multus-additional-cni-plugins-lphh6\" (UID: \"1dcac962-5867-4644-b3b3-ce62b3216b79\") " pod="openshift-multus/multus-additional-cni-plugins-lphh6"
May 25 13:05:50.026157 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.025896    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d21bdf8b-d0ab-48f2-8736-54e92947dd53-host-run-ovn-kubernetes\") pod \"ovnkube-node-vqnvc\" (UID: \"d21bdf8b-d0ab-48f2-8736-54e92947dd53\") " pod="openshift-ovn-kubernetes/ovnkube-node-vqnvc"
May 25 13:05:50.026157 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.025915    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b3857de1-9f56-4063-b132-410d217c9f8b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jc4n4\" (UID: \"b3857de1-9f56-4063-b132-410d217c9f8b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jc4n4"
May 25 13:05:50.026157 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.025931    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/87942b76-e2a7-407e-9a99-2ea92f737157-cni-binary-copy\") pod \"multus-scw67\" (UID: \"87942b76-e2a7-407e-9a99-2ea92f737157\") " pod="openshift-multus/multus-scw67"
May 25 13:05:50.026157 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.025965    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/87942b76-e2a7-407e-9a99-2ea92f737157-host-run-k8s-cni-cncf-io\") pod \"multus-scw67\" (UID: \"87942b76-e2a7-407e-9a99-2ea92f737157\") " pod="openshift-multus/multus-scw67"
May 25 13:05:50.026157 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.025985    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d21bdf8b-d0ab-48f2-8736-54e92947dd53-node-log\") pod \"ovnkube-node-vqnvc\" (UID: \"d21bdf8b-d0ab-48f2-8736-54e92947dd53\") " pod="openshift-ovn-kubernetes/ovnkube-node-vqnvc"
May 25 13:05:50.026157 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.025999    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d21bdf8b-d0ab-48f2-8736-54e92947dd53-ovn-node-metrics-cert\") pod \"ovnkube-node-vqnvc\" (UID: \"d21bdf8b-d0ab-48f2-8736-54e92947dd53\") " pod="openshift-ovn-kubernetes/ovnkube-node-vqnvc"
May 25 13:05:50.026157 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.026014    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b3857de1-9f56-4063-b132-410d217c9f8b-registration-dir\") pod \"aws-ebs-csi-driver-node-jc4n4\" (UID: \"b3857de1-9f56-4063-b132-410d217c9f8b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jc4n4"
May 25 13:05:50.026157 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.026031    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/904a1760-12f0-49b4-a389-c71b39a99619-etc-sysctl-conf\") pod \"tuned-lcmng\" (UID: \"904a1760-12f0-49b4-a389-c71b39a99619\") " pod="openshift-cluster-node-tuning-operator/tuned-lcmng"
May 25 13:05:50.026793 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.026068    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/87942b76-e2a7-407e-9a99-2ea92f737157-cnibin\") pod \"multus-scw67\" (UID: \"87942b76-e2a7-407e-9a99-2ea92f737157\") " pod="openshift-multus/multus-scw67"
May 25 13:05:50.026793 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.026091    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/55e3ea80-f133-45b9-b3b6-0d1abdbb1ad0-agent-certs\") pod \"konnectivity-agent-vwnkg\" (UID: \"55e3ea80-f133-45b9-b3b6-0d1abdbb1ad0\") " pod="kube-system/konnectivity-agent-vwnkg"
May 25 13:05:50.026793 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.026107    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/55e3ea80-f133-45b9-b3b6-0d1abdbb1ad0-konnectivity-ca\") pod \"konnectivity-agent-vwnkg\" (UID: \"55e3ea80-f133-45b9-b3b6-0d1abdbb1ad0\") " pod="kube-system/konnectivity-agent-vwnkg"
May 25 13:05:50.026793 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.026127    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d21bdf8b-d0ab-48f2-8736-54e92947dd53-etc-openvswitch\") pod \"ovnkube-node-vqnvc\" (UID: \"d21bdf8b-d0ab-48f2-8736-54e92947dd53\") " pod="openshift-ovn-kubernetes/ovnkube-node-vqnvc"
May 25 13:05:50.026793 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.026150    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/904a1760-12f0-49b4-a389-c71b39a99619-tmp\") pod \"tuned-lcmng\" (UID: \"904a1760-12f0-49b4-a389-c71b39a99619\") " pod="openshift-cluster-node-tuning-operator/tuned-lcmng"
May 25 13:05:50.026793 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.026173    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/87942b76-e2a7-407e-9a99-2ea92f737157-host-var-lib-kubelet\") pod \"multus-scw67\" (UID: \"87942b76-e2a7-407e-9a99-2ea92f737157\") " pod="openshift-multus/multus-scw67"
May 25 13:05:50.026793 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.026195    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b3857de1-9f56-4063-b132-410d217c9f8b-sys-fs\") pod \"aws-ebs-csi-driver-node-jc4n4\" (UID: \"b3857de1-9f56-4063-b132-410d217c9f8b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jc4n4"
May 25 13:05:50.026793 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.026216    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/904a1760-12f0-49b4-a389-c71b39a99619-var-lib-kubelet\") pod \"tuned-lcmng\" (UID: \"904a1760-12f0-49b4-a389-c71b39a99619\") " pod="openshift-cluster-node-tuning-operator/tuned-lcmng"
May 25 13:05:50.026793 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.026238    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8r6b\" (UniqueName: \"kubernetes.io/projected/8c2f1987-8965-449c-b426-b46515332392-kube-api-access-j8r6b\") pod \"network-metrics-daemon-5lm4j\" (UID: \"8c2f1987-8965-449c-b426-b46515332392\") " pod="openshift-multus/network-metrics-daemon-5lm4j"
May 25 13:05:50.026793 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.026254    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d21bdf8b-d0ab-48f2-8736-54e92947dd53-log-socket\") pod \"ovnkube-node-vqnvc\" (UID: \"d21bdf8b-d0ab-48f2-8736-54e92947dd53\") " pod="openshift-ovn-kubernetes/ovnkube-node-vqnvc"
May 25 13:05:50.026793 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.026270    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d21bdf8b-d0ab-48f2-8736-54e92947dd53-env-overrides\") pod \"ovnkube-node-vqnvc\" (UID: \"d21bdf8b-d0ab-48f2-8736-54e92947dd53\") " pod="openshift-ovn-kubernetes/ovnkube-node-vqnvc"
May 25 13:05:50.026793 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.026292    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1dcac962-5867-4644-b3b3-ce62b3216b79-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lphh6\" (UID: \"1dcac962-5867-4644-b3b3-ce62b3216b79\") " pod="openshift-multus/multus-additional-cni-plugins-lphh6"
May 25 13:05:50.026793 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.026314    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b3857de1-9f56-4063-b132-410d217c9f8b-etc-selinux\") pod \"aws-ebs-csi-driver-node-jc4n4\" (UID: \"b3857de1-9f56-4063-b132-410d217c9f8b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jc4n4"
May 25 13:05:50.026793 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.026329    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6f5ql\" (UniqueName: \"kubernetes.io/projected/71fb8f1c-14f2-441f-bbbf-92696b6559af-kube-api-access-6f5ql\") pod \"iptables-alerter-bwz4h\" (UID: \"71fb8f1c-14f2-441f-bbbf-92696b6559af\") " pod="openshift-network-operator/iptables-alerter-bwz4h"
May 25 13:05:50.026793 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.026383    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/904a1760-12f0-49b4-a389-c71b39a99619-etc-sysctl-d\") pod \"tuned-lcmng\" (UID: \"904a1760-12f0-49b4-a389-c71b39a99619\") " pod="openshift-cluster-node-tuning-operator/tuned-lcmng"
May 25 13:05:50.026793 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.026406    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/87942b76-e2a7-407e-9a99-2ea92f737157-multus-socket-dir-parent\") pod \"multus-scw67\" (UID: \"87942b76-e2a7-407e-9a99-2ea92f737157\") " pod="openshift-multus/multus-scw67"
May 25 13:05:50.027367 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.026429    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1dcac962-5867-4644-b3b3-ce62b3216b79-cni-binary-copy\") pod \"multus-additional-cni-plugins-lphh6\" (UID: \"1dcac962-5867-4644-b3b3-ce62b3216b79\") " pod="openshift-multus/multus-additional-cni-plugins-lphh6"
May 25 13:05:50.027367 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.026471    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d21bdf8b-d0ab-48f2-8736-54e92947dd53-run-systemd\") pod \"ovnkube-node-vqnvc\" (UID: \"d21bdf8b-d0ab-48f2-8736-54e92947dd53\") " pod="openshift-ovn-kubernetes/ovnkube-node-vqnvc"
May 25 13:05:50.027367 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.026502    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/904a1760-12f0-49b4-a389-c71b39a99619-etc-modprobe-d\") pod \"tuned-lcmng\" (UID: \"904a1760-12f0-49b4-a389-c71b39a99619\") " pod="openshift-cluster-node-tuning-operator/tuned-lcmng"
May 25 13:05:50.027367 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.026518    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/904a1760-12f0-49b4-a389-c71b39a99619-lib-modules\") pod \"tuned-lcmng\" (UID: \"904a1760-12f0-49b4-a389-c71b39a99619\") " pod="openshift-cluster-node-tuning-operator/tuned-lcmng"
May 25 13:05:50.027367 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.026535    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/87942b76-e2a7-407e-9a99-2ea92f737157-multus-cni-dir\") pod \"multus-scw67\" (UID: \"87942b76-e2a7-407e-9a99-2ea92f737157\") " pod="openshift-multus/multus-scw67"
May 25 13:05:50.027367 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.026550    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/87942b76-e2a7-407e-9a99-2ea92f737157-hostroot\") pod \"multus-scw67\" (UID: \"87942b76-e2a7-407e-9a99-2ea92f737157\") " pod="openshift-multus/multus-scw67"
May 25 13:05:50.027367 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.026565    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/87942b76-e2a7-407e-9a99-2ea92f737157-host-run-multus-certs\") pod \"multus-scw67\" (UID: \"87942b76-e2a7-407e-9a99-2ea92f737157\") " pod="openshift-multus/multus-scw67"
May 25 13:05:50.027367 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.026589    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/87942b76-e2a7-407e-9a99-2ea92f737157-etc-kubernetes\") pod \"multus-scw67\" (UID: \"87942b76-e2a7-407e-9a99-2ea92f737157\") " pod="openshift-multus/multus-scw67"
May 25 13:05:50.027367 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.026603    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9d1a5d53-d761-478d-a85e-0974f65121e3-host\") pod \"node-ca-v9wvs\" (UID: \"9d1a5d53-d761-478d-a85e-0974f65121e3\") " pod="openshift-image-registry/node-ca-v9wvs"
May 25 13:05:50.027367 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.026632    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1dcac962-5867-4644-b3b3-ce62b3216b79-cnibin\") pod \"multus-additional-cni-plugins-lphh6\" (UID: \"1dcac962-5867-4644-b3b3-ce62b3216b79\") " pod="openshift-multus/multus-additional-cni-plugins-lphh6"
May 25 13:05:50.027367 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.026651    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d21bdf8b-d0ab-48f2-8736-54e92947dd53-systemd-units\") pod \"ovnkube-node-vqnvc\" (UID: \"d21bdf8b-d0ab-48f2-8736-54e92947dd53\") " pod="openshift-ovn-kubernetes/ovnkube-node-vqnvc"
May 25 13:05:50.027367 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.026667    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d21bdf8b-d0ab-48f2-8736-54e92947dd53-var-lib-openvswitch\") pod \"ovnkube-node-vqnvc\" (UID: \"d21bdf8b-d0ab-48f2-8736-54e92947dd53\") " pod="openshift-ovn-kubernetes/ovnkube-node-vqnvc"
May 25 13:05:50.027367 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.026681    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d21bdf8b-d0ab-48f2-8736-54e92947dd53-host-cni-bin\") pod \"ovnkube-node-vqnvc\" (UID: \"d21bdf8b-d0ab-48f2-8736-54e92947dd53\") " pod="openshift-ovn-kubernetes/ovnkube-node-vqnvc"
May 25 13:05:50.027367 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.026701    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d21bdf8b-d0ab-48f2-8736-54e92947dd53-ovnkube-config\") pod \"ovnkube-node-vqnvc\" (UID: \"d21bdf8b-d0ab-48f2-8736-54e92947dd53\") " pod="openshift-ovn-kubernetes/ovnkube-node-vqnvc"
May 25 13:05:50.027367 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.026723    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/904a1760-12f0-49b4-a389-c71b39a99619-etc-tuned\") pod \"tuned-lcmng\" (UID: \"904a1760-12f0-49b4-a389-c71b39a99619\") " pod="openshift-cluster-node-tuning-operator/tuned-lcmng"
May 25 13:05:50.027367 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.026744    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/71fb8f1c-14f2-441f-bbbf-92696b6559af-iptables-alerter-script\") pod \"iptables-alerter-bwz4h\" (UID: \"71fb8f1c-14f2-441f-bbbf-92696b6559af\") " pod="openshift-network-operator/iptables-alerter-bwz4h"
May 25 13:05:50.027367 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.026771    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b3857de1-9f56-4063-b132-410d217c9f8b-socket-dir\") pod \"aws-ebs-csi-driver-node-jc4n4\" (UID: \"b3857de1-9f56-4063-b132-410d217c9f8b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jc4n4"
May 25 13:05:50.027967 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.026793    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b3857de1-9f56-4063-b132-410d217c9f8b-device-dir\") pod \"aws-ebs-csi-driver-node-jc4n4\" (UID: \"b3857de1-9f56-4063-b132-410d217c9f8b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jc4n4"
May 25 13:05:50.027967 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.026832    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1dcac962-5867-4644-b3b3-ce62b3216b79-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-lphh6\" (UID: \"1dcac962-5867-4644-b3b3-ce62b3216b79\") " pod="openshift-multus/multus-additional-cni-plugins-lphh6"
May 25 13:05:50.059907 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.059878    2557 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-05-24 13:00:49 +0000 UTC" deadline="2027-11-09 02:25:08.795109731 +0000 UTC"
May 25 13:05:50.059907 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.059904    2557 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12781h19m18.73520763s"
May 25 13:05:50.127409 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.127375    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/87942b76-e2a7-407e-9a99-2ea92f737157-os-release\") pod \"multus-scw67\" (UID: \"87942b76-e2a7-407e-9a99-2ea92f737157\") " pod="openshift-multus/multus-scw67"
May 25 13:05:50.127409 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.127416    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9d1a5d53-d761-478d-a85e-0974f65121e3-serviceca\") pod \"node-ca-v9wvs\" (UID: \"9d1a5d53-d761-478d-a85e-0974f65121e3\") " pod="openshift-image-registry/node-ca-v9wvs"
May 25 13:05:50.127646 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.127442    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c2f1987-8965-449c-b426-b46515332392-metrics-certs\") pod \"network-metrics-daemon-5lm4j\" (UID: \"8c2f1987-8965-449c-b426-b46515332392\") " pod="openshift-multus/network-metrics-daemon-5lm4j"
May 25 13:05:50.127646 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.127467    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d21bdf8b-d0ab-48f2-8736-54e92947dd53-run-openvswitch\") pod \"ovnkube-node-vqnvc\" (UID: \"d21bdf8b-d0ab-48f2-8736-54e92947dd53\") " pod="openshift-ovn-kubernetes/ovnkube-node-vqnvc"
May 25 13:05:50.127646 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.127518    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d21bdf8b-d0ab-48f2-8736-54e92947dd53-run-openvswitch\") pod \"ovnkube-node-vqnvc\" (UID: \"d21bdf8b-d0ab-48f2-8736-54e92947dd53\") " pod="openshift-ovn-kubernetes/ovnkube-node-vqnvc"
May 25 13:05:50.127646 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.127525    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/87942b76-e2a7-407e-9a99-2ea92f737157-os-release\") pod \"multus-scw67\" (UID: \"87942b76-e2a7-407e-9a99-2ea92f737157\") " pod="openshift-multus/multus-scw67"
May 25 13:05:50.127646 ip-10-0-143-235 kubenswrapper[2557]: E0525 13:05:50.127549    2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 13:05:50.127646 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.127575    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/71fb8f1c-14f2-441f-bbbf-92696b6559af-host-slash\") pod \"iptables-alerter-bwz4h\" (UID: \"71fb8f1c-14f2-441f-bbbf-92696b6559af\") " pod="openshift-network-operator/iptables-alerter-bwz4h"
May 25 13:05:50.127646 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.127642    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/71fb8f1c-14f2-441f-bbbf-92696b6559af-host-slash\") pod \"iptables-alerter-bwz4h\" (UID: \"71fb8f1c-14f2-441f-bbbf-92696b6559af\") " pod="openshift-network-operator/iptables-alerter-bwz4h"
May 25 13:05:50.127930 ip-10-0-143-235 kubenswrapper[2557]: E0525 13:05:50.127684    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c2f1987-8965-449c-b426-b46515332392-metrics-certs podName:8c2f1987-8965-449c-b426-b46515332392 nodeName:}" failed. No retries permitted until 2026-05-25 13:05:50.62761415 +0000 UTC m=+3.090630306 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8c2f1987-8965-449c-b426-b46515332392-metrics-certs") pod "network-metrics-daemon-5lm4j" (UID: "8c2f1987-8965-449c-b426-b46515332392") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 13:05:50.127930 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.127708    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/87942b76-e2a7-407e-9a99-2ea92f737157-host-var-lib-cni-bin\") pod \"multus-scw67\" (UID: \"87942b76-e2a7-407e-9a99-2ea92f737157\") " pod="openshift-multus/multus-scw67"
May 25 13:05:50.127930 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.127740    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-bk5mv\" (UniqueName: \"kubernetes.io/projected/9d1a5d53-d761-478d-a85e-0974f65121e3-kube-api-access-bk5mv\") pod \"node-ca-v9wvs\" (UID: \"9d1a5d53-d761-478d-a85e-0974f65121e3\") " pod="openshift-image-registry/node-ca-v9wvs"
May 25 13:05:50.127930 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.127742    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/87942b76-e2a7-407e-9a99-2ea92f737157-host-var-lib-cni-bin\") pod \"multus-scw67\" (UID: \"87942b76-e2a7-407e-9a99-2ea92f737157\") " pod="openshift-multus/multus-scw67"
May 25 13:05:50.127930 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.127767    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1dcac962-5867-4644-b3b3-ce62b3216b79-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-lphh6\" (UID: \"1dcac962-5867-4644-b3b3-ce62b3216b79\") " pod="openshift-multus/multus-additional-cni-plugins-lphh6"
May 25 13:05:50.128143 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.127953    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d21bdf8b-d0ab-48f2-8736-54e92947dd53-host-slash\") pod \"ovnkube-node-vqnvc\" (UID: \"d21bdf8b-d0ab-48f2-8736-54e92947dd53\") " pod="openshift-ovn-kubernetes/ovnkube-node-vqnvc"
May 25 13:05:50.128143 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.127987    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-pb9vv\" (UniqueName: \"kubernetes.io/projected/b3857de1-9f56-4063-b132-410d217c9f8b-kube-api-access-pb9vv\") pod \"aws-ebs-csi-driver-node-jc4n4\" (UID: \"b3857de1-9f56-4063-b132-410d217c9f8b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jc4n4"
May 25 13:05:50.128143 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.128000    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d21bdf8b-d0ab-48f2-8736-54e92947dd53-host-slash\") pod \"ovnkube-node-vqnvc\" (UID: \"d21bdf8b-d0ab-48f2-8736-54e92947dd53\") " pod="openshift-ovn-kubernetes/ovnkube-node-vqnvc"
May 25 13:05:50.128143 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.127954    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9d1a5d53-d761-478d-a85e-0974f65121e3-serviceca\") pod \"node-ca-v9wvs\" (UID: \"9d1a5d53-d761-478d-a85e-0974f65121e3\") " pod="openshift-image-registry/node-ca-v9wvs"
May 25 13:05:50.128143 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.128041    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-4f7rk\" (UniqueName: \"kubernetes.io/projected/a8d6074e-8880-4fd2-b612-de33be9d873e-kube-api-access-4f7rk\") pod \"network-check-target-96mr4\" (UID: \"a8d6074e-8880-4fd2-b612-de33be9d873e\") " pod="openshift-network-diagnostics/network-check-target-96mr4"
May 25 13:05:50.128143 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.128083    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/1dcac962-5867-4644-b3b3-ce62b3216b79-os-release\") pod \"multus-additional-cni-plugins-lphh6\" (UID: \"1dcac962-5867-4644-b3b3-ce62b3216b79\") " pod="openshift-multus/multus-additional-cni-plugins-lphh6"
May 25 13:05:50.128143 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.128107    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d21bdf8b-d0ab-48f2-8736-54e92947dd53-host-run-ovn-kubernetes\") pod \"ovnkube-node-vqnvc\" (UID: \"d21bdf8b-d0ab-48f2-8736-54e92947dd53\") " pod="openshift-ovn-kubernetes/ovnkube-node-vqnvc"
May 25 13:05:50.128143 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.128143    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b3857de1-9f56-4063-b132-410d217c9f8b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jc4n4\" (UID: \"b3857de1-9f56-4063-b132-410d217c9f8b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jc4n4"
May 25 13:05:50.128479 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.128166    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/87942b76-e2a7-407e-9a99-2ea92f737157-cni-binary-copy\") pod \"multus-scw67\" (UID: \"87942b76-e2a7-407e-9a99-2ea92f737157\") " pod="openshift-multus/multus-scw67"
May 25 13:05:50.128479 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.128217    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/87942b76-e2a7-407e-9a99-2ea92f737157-host-run-k8s-cni-cncf-io\") pod \"multus-scw67\" (UID: \"87942b76-e2a7-407e-9a99-2ea92f737157\") " pod="openshift-multus/multus-scw67"
May 25 13:05:50.128711 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.128690    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d21bdf8b-d0ab-48f2-8736-54e92947dd53-node-log\") pod \"ovnkube-node-vqnvc\" (UID: \"d21bdf8b-d0ab-48f2-8736-54e92947dd53\") " pod="openshift-ovn-kubernetes/ovnkube-node-vqnvc"
May 25 13:05:50.128806 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.128737    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d21bdf8b-d0ab-48f2-8736-54e92947dd53-ovn-node-metrics-cert\") pod \"ovnkube-node-vqnvc\" (UID: \"d21bdf8b-d0ab-48f2-8736-54e92947dd53\") " pod="openshift-ovn-kubernetes/ovnkube-node-vqnvc"
May 25 13:05:50.128806 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.128737    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1dcac962-5867-4644-b3b3-ce62b3216b79-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-lphh6\" (UID: \"1dcac962-5867-4644-b3b3-ce62b3216b79\") " pod="openshift-multus/multus-additional-cni-plugins-lphh6"
May 25 13:05:50.128806 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.128769    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b3857de1-9f56-4063-b132-410d217c9f8b-registration-dir\") pod \"aws-ebs-csi-driver-node-jc4n4\" (UID: \"b3857de1-9f56-4063-b132-410d217c9f8b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jc4n4"
May 25 13:05:50.128954 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.128895    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b3857de1-9f56-4063-b132-410d217c9f8b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jc4n4\" (UID: \"b3857de1-9f56-4063-b132-410d217c9f8b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jc4n4"
May 25 13:05:50.128954 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.128931    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d21bdf8b-d0ab-48f2-8736-54e92947dd53-host-run-ovn-kubernetes\") pod \"ovnkube-node-vqnvc\" (UID: \"d21bdf8b-d0ab-48f2-8736-54e92947dd53\") " pod="openshift-ovn-kubernetes/ovnkube-node-vqnvc"
May 25 13:05:50.128954 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.128949    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/904a1760-12f0-49b4-a389-c71b39a99619-etc-sysctl-conf\") pod \"tuned-lcmng\" (UID: \"904a1760-12f0-49b4-a389-c71b39a99619\") " pod="openshift-cluster-node-tuning-operator/tuned-lcmng"
May 25 13:05:50.129096 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.129031    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d21bdf8b-d0ab-48f2-8736-54e92947dd53-node-log\") pod \"ovnkube-node-vqnvc\" (UID: \"d21bdf8b-d0ab-48f2-8736-54e92947dd53\") " pod="openshift-ovn-kubernetes/ovnkube-node-vqnvc"
May 25 13:05:50.129096 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.129073    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b3857de1-9f56-4063-b132-410d217c9f8b-registration-dir\") pod \"aws-ebs-csi-driver-node-jc4n4\" (UID: \"b3857de1-9f56-4063-b132-410d217c9f8b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jc4n4"
May 25 13:05:50.129190 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.129108    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/1dcac962-5867-4644-b3b3-ce62b3216b79-os-release\") pod \"multus-additional-cni-plugins-lphh6\" (UID: \"1dcac962-5867-4644-b3b3-ce62b3216b79\") " pod="openshift-multus/multus-additional-cni-plugins-lphh6"
May 25 13:05:50.129791 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.129306    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/87942b76-e2a7-407e-9a99-2ea92f737157-host-run-k8s-cni-cncf-io\") pod \"multus-scw67\" (UID: \"87942b76-e2a7-407e-9a99-2ea92f737157\") " pod="openshift-multus/multus-scw67"
May 25 13:05:50.129791 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.128799    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/904a1760-12f0-49b4-a389-c71b39a99619-etc-sysctl-conf\") pod \"tuned-lcmng\" (UID: \"904a1760-12f0-49b4-a389-c71b39a99619\") " pod="openshift-cluster-node-tuning-operator/tuned-lcmng"
May 25 13:05:50.129791 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.129559    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/87942b76-e2a7-407e-9a99-2ea92f737157-cnibin\") pod \"multus-scw67\" (UID: \"87942b76-e2a7-407e-9a99-2ea92f737157\") " pod="openshift-multus/multus-scw67"
May 25 13:05:50.129791 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.129640    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/55e3ea80-f133-45b9-b3b6-0d1abdbb1ad0-agent-certs\") pod \"konnectivity-agent-vwnkg\" (UID: \"55e3ea80-f133-45b9-b3b6-0d1abdbb1ad0\") " pod="kube-system/konnectivity-agent-vwnkg"
May 25 13:05:50.129791 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.129680    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/55e3ea80-f133-45b9-b3b6-0d1abdbb1ad0-konnectivity-ca\") pod \"konnectivity-agent-vwnkg\" (UID: \"55e3ea80-f133-45b9-b3b6-0d1abdbb1ad0\") " pod="kube-system/konnectivity-agent-vwnkg"
May 25 13:05:50.129791 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.129702    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/87942b76-e2a7-407e-9a99-2ea92f737157-cni-binary-copy\") pod \"multus-scw67\" (UID: \"87942b76-e2a7-407e-9a99-2ea92f737157\") " pod="openshift-multus/multus-scw67"
May 25 13:05:50.129791 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.129726    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d21bdf8b-d0ab-48f2-8736-54e92947dd53-etc-openvswitch\") pod \"ovnkube-node-vqnvc\" (UID: \"d21bdf8b-d0ab-48f2-8736-54e92947dd53\") " pod="openshift-ovn-kubernetes/ovnkube-node-vqnvc"
May 25 13:05:50.129791 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.129782    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/904a1760-12f0-49b4-a389-c71b39a99619-tmp\") pod \"tuned-lcmng\" (UID: \"904a1760-12f0-49b4-a389-c71b39a99619\") " pod="openshift-cluster-node-tuning-operator/tuned-lcmng"
May 25 13:05:50.130217 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.129821    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/87942b76-e2a7-407e-9a99-2ea92f737157-host-var-lib-kubelet\") pod \"multus-scw67\" (UID: \"87942b76-e2a7-407e-9a99-2ea92f737157\") " pod="openshift-multus/multus-scw67"
May 25 13:05:50.130217 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.129856    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/87942b76-e2a7-407e-9a99-2ea92f737157-cnibin\") pod \"multus-scw67\" (UID: \"87942b76-e2a7-407e-9a99-2ea92f737157\") " pod="openshift-multus/multus-scw67"
May 25 13:05:50.130217 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.129873    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b3857de1-9f56-4063-b132-410d217c9f8b-sys-fs\") pod \"aws-ebs-csi-driver-node-jc4n4\" (UID: \"b3857de1-9f56-4063-b132-410d217c9f8b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jc4n4"
May 25 13:05:50.130217 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.129925    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/904a1760-12f0-49b4-a389-c71b39a99619-var-lib-kubelet\") pod \"tuned-lcmng\" (UID: \"904a1760-12f0-49b4-a389-c71b39a99619\") " pod="openshift-cluster-node-tuning-operator/tuned-lcmng"
May 25 13:05:50.130217 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.130000    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-j8r6b\" (UniqueName: \"kubernetes.io/projected/8c2f1987-8965-449c-b426-b46515332392-kube-api-access-j8r6b\") pod \"network-metrics-daemon-5lm4j\" (UID: \"8c2f1987-8965-449c-b426-b46515332392\") " pod="openshift-multus/network-metrics-daemon-5lm4j"
May 25 13:05:50.130217 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.130022    2557 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
May 25 13:05:50.130217 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.130040    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d21bdf8b-d0ab-48f2-8736-54e92947dd53-log-socket\") pod \"ovnkube-node-vqnvc\" (UID: \"d21bdf8b-d0ab-48f2-8736-54e92947dd53\") " pod="openshift-ovn-kubernetes/ovnkube-node-vqnvc"
May 25 13:05:50.130217 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.130047    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/87942b76-e2a7-407e-9a99-2ea92f737157-host-var-lib-kubelet\") pod \"multus-scw67\" (UID: \"87942b76-e2a7-407e-9a99-2ea92f737157\") " pod="openshift-multus/multus-scw67"
May 25 13:05:50.130217 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.130074    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d21bdf8b-d0ab-48f2-8736-54e92947dd53-env-overrides\") pod \"ovnkube-node-vqnvc\" (UID: \"d21bdf8b-d0ab-48f2-8736-54e92947dd53\") " pod="openshift-ovn-kubernetes/ovnkube-node-vqnvc"
May 25 13:05:50.130217 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.130138    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1dcac962-5867-4644-b3b3-ce62b3216b79-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lphh6\" (UID: \"1dcac962-5867-4644-b3b3-ce62b3216b79\") " pod="openshift-multus/multus-additional-cni-plugins-lphh6"
May 25 13:05:50.130217 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.130184    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b3857de1-9f56-4063-b132-410d217c9f8b-etc-selinux\") pod \"aws-ebs-csi-driver-node-jc4n4\" (UID: \"b3857de1-9f56-4063-b132-410d217c9f8b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jc4n4"
May 25 13:05:50.130217 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.130221    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-6f5ql\" (UniqueName: \"kubernetes.io/projected/71fb8f1c-14f2-441f-bbbf-92696b6559af-kube-api-access-6f5ql\") pod \"iptables-alerter-bwz4h\" (UID: \"71fb8f1c-14f2-441f-bbbf-92696b6559af\") " pod="openshift-network-operator/iptables-alerter-bwz4h"
May 25 13:05:50.130753 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.130261    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/904a1760-12f0-49b4-a389-c71b39a99619-etc-sysctl-d\") pod \"tuned-lcmng\" (UID: \"904a1760-12f0-49b4-a389-c71b39a99619\") " pod="openshift-cluster-node-tuning-operator/tuned-lcmng"
May 25 13:05:50.130753 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.130297    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/87942b76-e2a7-407e-9a99-2ea92f737157-multus-socket-dir-parent\") pod \"multus-scw67\" (UID: \"87942b76-e2a7-407e-9a99-2ea92f737157\") " pod="openshift-multus/multus-scw67"
May 25 13:05:50.130753 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.130383    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1dcac962-5867-4644-b3b3-ce62b3216b79-cni-binary-copy\") pod \"multus-additional-cni-plugins-lphh6\" (UID: \"1dcac962-5867-4644-b3b3-ce62b3216b79\") " pod="openshift-multus/multus-additional-cni-plugins-lphh6"
May 25 13:05:50.130753 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.130459    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d21bdf8b-d0ab-48f2-8736-54e92947dd53-log-socket\") pod \"ovnkube-node-vqnvc\" (UID: \"d21bdf8b-d0ab-48f2-8736-54e92947dd53\") " pod="openshift-ovn-kubernetes/ovnkube-node-vqnvc"
May 25 13:05:50.130753 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.130604    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b3857de1-9f56-4063-b132-410d217c9f8b-sys-fs\") pod \"aws-ebs-csi-driver-node-jc4n4\" (UID: \"b3857de1-9f56-4063-b132-410d217c9f8b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jc4n4"
May 25 13:05:50.130753 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.130656    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d21bdf8b-d0ab-48f2-8736-54e92947dd53-run-systemd\") pod \"ovnkube-node-vqnvc\" (UID: \"d21bdf8b-d0ab-48f2-8736-54e92947dd53\") " pod="openshift-ovn-kubernetes/ovnkube-node-vqnvc"
May 25 13:05:50.130753 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.130670    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/904a1760-12f0-49b4-a389-c71b39a99619-var-lib-kubelet\") pod \"tuned-lcmng\" (UID: \"904a1760-12f0-49b4-a389-c71b39a99619\") " pod="openshift-cluster-node-tuning-operator/tuned-lcmng"
May 25 13:05:50.130753 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.130691    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/904a1760-12f0-49b4-a389-c71b39a99619-etc-modprobe-d\") pod \"tuned-lcmng\" (UID: \"904a1760-12f0-49b4-a389-c71b39a99619\") " pod="openshift-cluster-node-tuning-operator/tuned-lcmng"
May 25 13:05:50.130753 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.130721    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/904a1760-12f0-49b4-a389-c71b39a99619-lib-modules\") pod \"tuned-lcmng\" (UID: \"904a1760-12f0-49b4-a389-c71b39a99619\") " pod="openshift-cluster-node-tuning-operator/tuned-lcmng"
May 25 13:05:50.131153 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.130769    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/87942b76-e2a7-407e-9a99-2ea92f737157-multus-cni-dir\") pod \"multus-scw67\" (UID: \"87942b76-e2a7-407e-9a99-2ea92f737157\") " pod="openshift-multus/multus-scw67"
May 25 13:05:50.131612 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.131310    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/55e3ea80-f133-45b9-b3b6-0d1abdbb1ad0-konnectivity-ca\") pod \"konnectivity-agent-vwnkg\" (UID: \"55e3ea80-f133-45b9-b3b6-0d1abdbb1ad0\") " pod="kube-system/konnectivity-agent-vwnkg"
May 25 13:05:50.131612 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.131331    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d21bdf8b-d0ab-48f2-8736-54e92947dd53-env-overrides\") pod \"ovnkube-node-vqnvc\" (UID: \"d21bdf8b-d0ab-48f2-8736-54e92947dd53\") " pod="openshift-ovn-kubernetes/ovnkube-node-vqnvc"
May 25 13:05:50.131612 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.131369    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/87942b76-e2a7-407e-9a99-2ea92f737157-hostroot\") pod \"multus-scw67\" (UID: \"87942b76-e2a7-407e-9a99-2ea92f737157\") " pod="openshift-multus/multus-scw67"
May 25 13:05:50.131612 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.131410    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/87942b76-e2a7-407e-9a99-2ea92f737157-host-run-multus-certs\") pod \"multus-scw67\" (UID: \"87942b76-e2a7-407e-9a99-2ea92f737157\") " pod="openshift-multus/multus-scw67"
May 25 13:05:50.131612 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.131462    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/87942b76-e2a7-407e-9a99-2ea92f737157-etc-kubernetes\") pod \"multus-scw67\" (UID: \"87942b76-e2a7-407e-9a99-2ea92f737157\") " pod="openshift-multus/multus-scw67"
May 25 13:05:50.131612 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.131559    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/87942b76-e2a7-407e-9a99-2ea92f737157-multus-cni-dir\") pod \"multus-scw67\" (UID: \"87942b76-e2a7-407e-9a99-2ea92f737157\") " pod="openshift-multus/multus-scw67"
May 25 13:05:50.131612 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.131568    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/87942b76-e2a7-407e-9a99-2ea92f737157-hostroot\") pod \"multus-scw67\" (UID: \"87942b76-e2a7-407e-9a99-2ea92f737157\") " pod="openshift-multus/multus-scw67"
May 25 13:05:50.131612 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.131575    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1dcac962-5867-4644-b3b3-ce62b3216b79-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lphh6\" (UID: \"1dcac962-5867-4644-b3b3-ce62b3216b79\") " pod="openshift-multus/multus-additional-cni-plugins-lphh6"
May 25 13:05:50.132024 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.131629    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9d1a5d53-d761-478d-a85e-0974f65121e3-host\") pod \"node-ca-v9wvs\" (UID: \"9d1a5d53-d761-478d-a85e-0974f65121e3\") " pod="openshift-image-registry/node-ca-v9wvs"
May 25 13:05:50.132024 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.131654    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d21bdf8b-d0ab-48f2-8736-54e92947dd53-run-systemd\") pod \"ovnkube-node-vqnvc\" (UID: \"d21bdf8b-d0ab-48f2-8736-54e92947dd53\") " pod="openshift-ovn-kubernetes/ovnkube-node-vqnvc"
May 25 13:05:50.132024 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.131661    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1dcac962-5867-4644-b3b3-ce62b3216b79-cnibin\") pod \"multus-additional-cni-plugins-lphh6\" (UID: \"1dcac962-5867-4644-b3b3-ce62b3216b79\") " pod="openshift-multus/multus-additional-cni-plugins-lphh6"
May 25 13:05:50.132024 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.131715    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d21bdf8b-d0ab-48f2-8736-54e92947dd53-systemd-units\") pod \"ovnkube-node-vqnvc\" (UID: \"d21bdf8b-d0ab-48f2-8736-54e92947dd53\") " pod="openshift-ovn-kubernetes/ovnkube-node-vqnvc"
May 25 13:05:50.132024 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.131756    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/904a1760-12f0-49b4-a389-c71b39a99619-lib-modules\") pod \"tuned-lcmng\" (UID: \"904a1760-12f0-49b4-a389-c71b39a99619\") " pod="openshift-cluster-node-tuning-operator/tuned-lcmng"
May 25 13:05:50.132024 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.131750    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d21bdf8b-d0ab-48f2-8736-54e92947dd53-var-lib-openvswitch\") pod \"ovnkube-node-vqnvc\" (UID: \"d21bdf8b-d0ab-48f2-8736-54e92947dd53\") " pod="openshift-ovn-kubernetes/ovnkube-node-vqnvc"
May 25 13:05:50.132024 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.131846    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d21bdf8b-d0ab-48f2-8736-54e92947dd53-var-lib-openvswitch\") pod \"ovnkube-node-vqnvc\" (UID: \"d21bdf8b-d0ab-48f2-8736-54e92947dd53\") " pod="openshift-ovn-kubernetes/ovnkube-node-vqnvc"
May 25 13:05:50.132024 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.131906    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9d1a5d53-d761-478d-a85e-0974f65121e3-host\") pod \"node-ca-v9wvs\" (UID: \"9d1a5d53-d761-478d-a85e-0974f65121e3\") " pod="openshift-image-registry/node-ca-v9wvs"
May 25 13:05:50.132024 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.131937    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d21bdf8b-d0ab-48f2-8736-54e92947dd53-systemd-units\") pod \"ovnkube-node-vqnvc\" (UID: \"d21bdf8b-d0ab-48f2-8736-54e92947dd53\") " pod="openshift-ovn-kubernetes/ovnkube-node-vqnvc"
May 25 13:05:50.132024 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.131905    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/904a1760-12f0-49b4-a389-c71b39a99619-etc-sysctl-d\") pod \"tuned-lcmng\" (UID: \"904a1760-12f0-49b4-a389-c71b39a99619\") " pod="openshift-cluster-node-tuning-operator/tuned-lcmng"
May 25 13:05:50.132024 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.131964    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d21bdf8b-d0ab-48f2-8736-54e92947dd53-etc-openvswitch\") pod \"ovnkube-node-vqnvc\" (UID: \"d21bdf8b-d0ab-48f2-8736-54e92947dd53\") " pod="openshift-ovn-kubernetes/ovnkube-node-vqnvc"
May 25 13:05:50.132024 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.131998    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/904a1760-12f0-49b4-a389-c71b39a99619-etc-modprobe-d\") pod \"tuned-lcmng\" (UID: \"904a1760-12f0-49b4-a389-c71b39a99619\") " pod="openshift-cluster-node-tuning-operator/tuned-lcmng"
May 25 13:05:50.132636 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.132120    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b3857de1-9f56-4063-b132-410d217c9f8b-etc-selinux\") pod \"aws-ebs-csi-driver-node-jc4n4\" (UID: \"b3857de1-9f56-4063-b132-410d217c9f8b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jc4n4"
May 25 13:05:50.132636 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.132126    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1dcac962-5867-4644-b3b3-ce62b3216b79-cnibin\") pod \"multus-additional-cni-plugins-lphh6\" (UID: \"1dcac962-5867-4644-b3b3-ce62b3216b79\") " pod="openshift-multus/multus-additional-cni-plugins-lphh6"
May 25 13:05:50.132636 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.132173    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/87942b76-e2a7-407e-9a99-2ea92f737157-etc-kubernetes\") pod \"multus-scw67\" (UID: \"87942b76-e2a7-407e-9a99-2ea92f737157\") " pod="openshift-multus/multus-scw67"
May 25 13:05:50.132636 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.132245    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/87942b76-e2a7-407e-9a99-2ea92f737157-multus-socket-dir-parent\") pod \"multus-scw67\" (UID: \"87942b76-e2a7-407e-9a99-2ea92f737157\") " pod="openshift-multus/multus-scw67"
May 25 13:05:50.132636 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.132270    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d21bdf8b-d0ab-48f2-8736-54e92947dd53-host-cni-bin\") pod \"ovnkube-node-vqnvc\" (UID: \"d21bdf8b-d0ab-48f2-8736-54e92947dd53\") " pod="openshift-ovn-kubernetes/ovnkube-node-vqnvc"
May 25 13:05:50.132636 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.132302    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d21bdf8b-d0ab-48f2-8736-54e92947dd53-ovnkube-config\") pod \"ovnkube-node-vqnvc\" (UID: \"d21bdf8b-d0ab-48f2-8736-54e92947dd53\") " pod="openshift-ovn-kubernetes/ovnkube-node-vqnvc"
May 25 13:05:50.132636 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.132333    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/904a1760-12f0-49b4-a389-c71b39a99619-etc-tuned\") pod \"tuned-lcmng\" (UID: \"904a1760-12f0-49b4-a389-c71b39a99619\") " pod="openshift-cluster-node-tuning-operator/tuned-lcmng"
May 25 13:05:50.132636 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.132409    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d21bdf8b-d0ab-48f2-8736-54e92947dd53-host-cni-bin\") pod \"ovnkube-node-vqnvc\" (UID: \"d21bdf8b-d0ab-48f2-8736-54e92947dd53\") " pod="openshift-ovn-kubernetes/ovnkube-node-vqnvc"
May 25 13:05:50.132636 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.132495    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/71fb8f1c-14f2-441f-bbbf-92696b6559af-iptables-alerter-script\") pod \"iptables-alerter-bwz4h\" (UID: \"71fb8f1c-14f2-441f-bbbf-92696b6559af\") " pod="openshift-network-operator/iptables-alerter-bwz4h"
May 25 13:05:50.132636 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.132552    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b3857de1-9f56-4063-b132-410d217c9f8b-socket-dir\") pod \"aws-ebs-csi-driver-node-jc4n4\" (UID: \"b3857de1-9f56-4063-b132-410d217c9f8b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jc4n4"
May 25 13:05:50.132636 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.132606    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b3857de1-9f56-4063-b132-410d217c9f8b-device-dir\") pod \"aws-ebs-csi-driver-node-jc4n4\" (UID: \"b3857de1-9f56-4063-b132-410d217c9f8b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jc4n4"
May 25 13:05:50.133119 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.132647    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1dcac962-5867-4644-b3b3-ce62b3216b79-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-lphh6\" (UID: \"1dcac962-5867-4644-b3b3-ce62b3216b79\") " pod="openshift-multus/multus-additional-cni-plugins-lphh6"
May 25 13:05:50.133119 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.132681    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/904a1760-12f0-49b4-a389-c71b39a99619-etc-kubernetes\") pod \"tuned-lcmng\" (UID: \"904a1760-12f0-49b4-a389-c71b39a99619\") " pod="openshift-cluster-node-tuning-operator/tuned-lcmng"
May 25 13:05:50.133119 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.132717    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d21bdf8b-d0ab-48f2-8736-54e92947dd53-host-run-netns\") pod \"ovnkube-node-vqnvc\" (UID: \"d21bdf8b-d0ab-48f2-8736-54e92947dd53\") " pod="openshift-ovn-kubernetes/ovnkube-node-vqnvc"
May 25 13:05:50.133119 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.132749    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-96fp8\" (UniqueName: \"kubernetes.io/projected/d21bdf8b-d0ab-48f2-8736-54e92947dd53-kube-api-access-96fp8\") pod \"ovnkube-node-vqnvc\" (UID: \"d21bdf8b-d0ab-48f2-8736-54e92947dd53\") " pod="openshift-ovn-kubernetes/ovnkube-node-vqnvc"
May 25 13:05:50.133119 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.132775    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/904a1760-12f0-49b4-a389-c71b39a99619-etc-sysconfig\") pod \"tuned-lcmng\" (UID: \"904a1760-12f0-49b4-a389-c71b39a99619\") " pod="openshift-cluster-node-tuning-operator/tuned-lcmng"
May 25 13:05:50.133119 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.132805    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-kvzsd\" (UniqueName: \"kubernetes.io/projected/904a1760-12f0-49b4-a389-c71b39a99619-kube-api-access-kvzsd\") pod \"tuned-lcmng\" (UID: \"904a1760-12f0-49b4-a389-c71b39a99619\") " pod="openshift-cluster-node-tuning-operator/tuned-lcmng"
May 25 13:05:50.133119 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.132831    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/87942b76-e2a7-407e-9a99-2ea92f737157-multus-conf-dir\") pod \"multus-scw67\" (UID: \"87942b76-e2a7-407e-9a99-2ea92f737157\") " pod="openshift-multus/multus-scw67"
May 25 13:05:50.133119 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.132881    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1dcac962-5867-4644-b3b3-ce62b3216b79-cni-binary-copy\") pod \"multus-additional-cni-plugins-lphh6\" (UID: \"1dcac962-5867-4644-b3b3-ce62b3216b79\") " pod="openshift-multus/multus-additional-cni-plugins-lphh6"
May 25 13:05:50.133119 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.132893    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d21bdf8b-d0ab-48f2-8736-54e92947dd53-ovnkube-config\") pod \"ovnkube-node-vqnvc\" (UID: \"d21bdf8b-d0ab-48f2-8736-54e92947dd53\") " pod="openshift-ovn-kubernetes/ovnkube-node-vqnvc"
May 25 13:05:50.133572 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.133430    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1dcac962-5867-4644-b3b3-ce62b3216b79-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-lphh6\" (UID: \"1dcac962-5867-4644-b3b3-ce62b3216b79\") " pod="openshift-multus/multus-additional-cni-plugins-lphh6"
May 25 13:05:50.133572 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.133451    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/71fb8f1c-14f2-441f-bbbf-92696b6559af-iptables-alerter-script\") pod \"iptables-alerter-bwz4h\" (UID: \"71fb8f1c-14f2-441f-bbbf-92696b6559af\") " pod="openshift-network-operator/iptables-alerter-bwz4h"
May 25 13:05:50.133572 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.133538    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/904a1760-12f0-49b4-a389-c71b39a99619-etc-kubernetes\") pod \"tuned-lcmng\" (UID: \"904a1760-12f0-49b4-a389-c71b39a99619\") " pod="openshift-cluster-node-tuning-operator/tuned-lcmng"
May 25 13:05:50.134121 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.133784    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/904a1760-12f0-49b4-a389-c71b39a99619-etc-sysconfig\") pod \"tuned-lcmng\" (UID: \"904a1760-12f0-49b4-a389-c71b39a99619\") " pod="openshift-cluster-node-tuning-operator/tuned-lcmng"
May 25 13:05:50.134121 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.133807    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d21bdf8b-d0ab-48f2-8736-54e92947dd53-ovn-node-metrics-cert\") pod \"ovnkube-node-vqnvc\" (UID: \"d21bdf8b-d0ab-48f2-8736-54e92947dd53\") " pod="openshift-ovn-kubernetes/ovnkube-node-vqnvc"
May 25 13:05:50.134121 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.132886    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d21bdf8b-d0ab-48f2-8736-54e92947dd53-ovnkube-script-lib\") pod \"ovnkube-node-vqnvc\" (UID: \"d21bdf8b-d0ab-48f2-8736-54e92947dd53\") " pod="openshift-ovn-kubernetes/ovnkube-node-vqnvc"
May 25 13:05:50.134121 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.133864    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/904a1760-12f0-49b4-a389-c71b39a99619-host\") pod \"tuned-lcmng\" (UID: \"904a1760-12f0-49b4-a389-c71b39a99619\") " pod="openshift-cluster-node-tuning-operator/tuned-lcmng"
May 25 13:05:50.134121 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.133888    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d21bdf8b-d0ab-48f2-8736-54e92947dd53-host-run-netns\") pod \"ovnkube-node-vqnvc\" (UID: \"d21bdf8b-d0ab-48f2-8736-54e92947dd53\") " pod="openshift-ovn-kubernetes/ovnkube-node-vqnvc"
May 25 13:05:50.134121 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.133900    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/87942b76-e2a7-407e-9a99-2ea92f737157-host-run-netns\") pod \"multus-scw67\" (UID: \"87942b76-e2a7-407e-9a99-2ea92f737157\") " pod="openshift-multus/multus-scw67"
May 25 13:05:50.134121 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.133946    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/87942b76-e2a7-407e-9a99-2ea92f737157-host-run-netns\") pod \"multus-scw67\" (UID: \"87942b76-e2a7-407e-9a99-2ea92f737157\") " pod="openshift-multus/multus-scw67"
May 25 13:05:50.134121 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.133946    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d21bdf8b-d0ab-48f2-8736-54e92947dd53-host-kubelet\") pod \"ovnkube-node-vqnvc\" (UID: \"d21bdf8b-d0ab-48f2-8736-54e92947dd53\") " pod="openshift-ovn-kubernetes/ovnkube-node-vqnvc"
May 25 13:05:50.134121 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.133999    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/904a1760-12f0-49b4-a389-c71b39a99619-host\") pod \"tuned-lcmng\" (UID: \"904a1760-12f0-49b4-a389-c71b39a99619\") " pod="openshift-cluster-node-tuning-operator/tuned-lcmng"
May 25 13:05:50.134121 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.134000    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/904a1760-12f0-49b4-a389-c71b39a99619-sys\") pod \"tuned-lcmng\" (UID: \"904a1760-12f0-49b4-a389-c71b39a99619\") " pod="openshift-cluster-node-tuning-operator/tuned-lcmng"
May 25 13:05:50.134121 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.134043    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/87942b76-e2a7-407e-9a99-2ea92f737157-system-cni-dir\") pod \"multus-scw67\" (UID: \"87942b76-e2a7-407e-9a99-2ea92f737157\") " pod="openshift-multus/multus-scw67"
May 25 13:05:50.134121 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.134096    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/87942b76-e2a7-407e-9a99-2ea92f737157-host-var-lib-cni-multus\") pod \"multus-scw67\" (UID: \"87942b76-e2a7-407e-9a99-2ea92f737157\") " pod="openshift-multus/multus-scw67"
May 25 13:05:50.134709 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.134138    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-xx67l\" (UniqueName: \"kubernetes.io/projected/87942b76-e2a7-407e-9a99-2ea92f737157-kube-api-access-xx67l\") pod \"multus-scw67\" (UID: \"87942b76-e2a7-407e-9a99-2ea92f737157\") " pod="openshift-multus/multus-scw67"
May 25 13:05:50.134709 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.134175    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d21bdf8b-d0ab-48f2-8736-54e92947dd53-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vqnvc\" (UID: \"d21bdf8b-d0ab-48f2-8736-54e92947dd53\") " pod="openshift-ovn-kubernetes/ovnkube-node-vqnvc"
May 25 13:05:50.134709 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.134230    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/904a1760-12f0-49b4-a389-c71b39a99619-etc-systemd\") pod \"tuned-lcmng\" (UID: \"904a1760-12f0-49b4-a389-c71b39a99619\") " pod="openshift-cluster-node-tuning-operator/tuned-lcmng"
May 25 13:05:50.134709 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.134259    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/87942b76-e2a7-407e-9a99-2ea92f737157-multus-daemon-config\") pod \"multus-scw67\" (UID: \"87942b76-e2a7-407e-9a99-2ea92f737157\") " pod="openshift-multus/multus-scw67"
May 25 13:05:50.134709 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.134269    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/87942b76-e2a7-407e-9a99-2ea92f737157-system-cni-dir\") pod \"multus-scw67\" (UID: \"87942b76-e2a7-407e-9a99-2ea92f737157\") " pod="openshift-multus/multus-scw67"
May 25 13:05:50.134709 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.134323    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1dcac962-5867-4644-b3b3-ce62b3216b79-system-cni-dir\") pod \"multus-additional-cni-plugins-lphh6\" (UID: \"1dcac962-5867-4644-b3b3-ce62b3216b79\") " pod="openshift-multus/multus-additional-cni-plugins-lphh6"
May 25 13:05:50.134709 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.134362    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/87942b76-e2a7-407e-9a99-2ea92f737157-multus-conf-dir\") pod \"multus-scw67\" (UID: \"87942b76-e2a7-407e-9a99-2ea92f737157\") " pod="openshift-multus/multus-scw67"
May 25 13:05:50.134709 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.134374    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-w9d77\" (UniqueName: \"kubernetes.io/projected/1dcac962-5867-4644-b3b3-ce62b3216b79-kube-api-access-w9d77\") pod \"multus-additional-cni-plugins-lphh6\" (UID: \"1dcac962-5867-4644-b3b3-ce62b3216b79\") " pod="openshift-multus/multus-additional-cni-plugins-lphh6"
May 25 13:05:50.134709 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.134421    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b3857de1-9f56-4063-b132-410d217c9f8b-device-dir\") pod \"aws-ebs-csi-driver-node-jc4n4\" (UID: \"b3857de1-9f56-4063-b132-410d217c9f8b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jc4n4"
May 25 13:05:50.134709 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.134462    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d21bdf8b-d0ab-48f2-8736-54e92947dd53-run-ovn\") pod \"ovnkube-node-vqnvc\" (UID: \"d21bdf8b-d0ab-48f2-8736-54e92947dd53\") " pod="openshift-ovn-kubernetes/ovnkube-node-vqnvc"
May 25 13:05:50.134709 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.134503    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d21bdf8b-d0ab-48f2-8736-54e92947dd53-host-cni-netd\") pod \"ovnkube-node-vqnvc\" (UID: \"d21bdf8b-d0ab-48f2-8736-54e92947dd53\") " pod="openshift-ovn-kubernetes/ovnkube-node-vqnvc"
May 25 13:05:50.134709 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.134540    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/904a1760-12f0-49b4-a389-c71b39a99619-run\") pod \"tuned-lcmng\" (UID: \"904a1760-12f0-49b4-a389-c71b39a99619\") " pod="openshift-cluster-node-tuning-operator/tuned-lcmng"
May 25 13:05:50.134709 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.134671    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/904a1760-12f0-49b4-a389-c71b39a99619-run\") pod \"tuned-lcmng\" (UID: \"904a1760-12f0-49b4-a389-c71b39a99619\") " pod="openshift-cluster-node-tuning-operator/tuned-lcmng"
May 25 13:05:50.135269 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.134729    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d21bdf8b-d0ab-48f2-8736-54e92947dd53-host-kubelet\") pod \"ovnkube-node-vqnvc\" (UID: \"d21bdf8b-d0ab-48f2-8736-54e92947dd53\") " pod="openshift-ovn-kubernetes/ovnkube-node-vqnvc"
May 25 13:05:50.135269 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.134750    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/87942b76-e2a7-407e-9a99-2ea92f737157-host-var-lib-cni-multus\") pod \"multus-scw67\" (UID: \"87942b76-e2a7-407e-9a99-2ea92f737157\") " pod="openshift-multus/multus-scw67"
May 25 13:05:50.135269 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.134790    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/904a1760-12f0-49b4-a389-c71b39a99619-sys\") pod \"tuned-lcmng\" (UID: \"904a1760-12f0-49b4-a389-c71b39a99619\") " pod="openshift-cluster-node-tuning-operator/tuned-lcmng"
May 25 13:05:50.135269 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.134850    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1dcac962-5867-4644-b3b3-ce62b3216b79-system-cni-dir\") pod \"multus-additional-cni-plugins-lphh6\" (UID: \"1dcac962-5867-4644-b3b3-ce62b3216b79\") " pod="openshift-multus/multus-additional-cni-plugins-lphh6"
May 25 13:05:50.135269 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.134856    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d21bdf8b-d0ab-48f2-8736-54e92947dd53-run-ovn\") pod \"ovnkube-node-vqnvc\" (UID: \"d21bdf8b-d0ab-48f2-8736-54e92947dd53\") " pod="openshift-ovn-kubernetes/ovnkube-node-vqnvc"
May 25 13:05:50.135269 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.134878    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d21bdf8b-d0ab-48f2-8736-54e92947dd53-ovnkube-script-lib\") pod \"ovnkube-node-vqnvc\" (UID: \"d21bdf8b-d0ab-48f2-8736-54e92947dd53\") " pod="openshift-ovn-kubernetes/ovnkube-node-vqnvc"
May 25 13:05:50.135269 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.134905    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d21bdf8b-d0ab-48f2-8736-54e92947dd53-host-cni-netd\") pod \"ovnkube-node-vqnvc\" (UID: \"d21bdf8b-d0ab-48f2-8736-54e92947dd53\") " pod="openshift-ovn-kubernetes/ovnkube-node-vqnvc"
May 25 13:05:50.135269 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.135076    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d21bdf8b-d0ab-48f2-8736-54e92947dd53-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vqnvc\" (UID: \"d21bdf8b-d0ab-48f2-8736-54e92947dd53\") " pod="openshift-ovn-kubernetes/ovnkube-node-vqnvc"
May 25 13:05:50.135600 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.135297    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b3857de1-9f56-4063-b132-410d217c9f8b-socket-dir\") pod \"aws-ebs-csi-driver-node-jc4n4\" (UID: \"b3857de1-9f56-4063-b132-410d217c9f8b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jc4n4"
May 25 13:05:50.135600 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.135320    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/87942b76-e2a7-407e-9a99-2ea92f737157-host-run-multus-certs\") pod \"multus-scw67\" (UID: \"87942b76-e2a7-407e-9a99-2ea92f737157\") " pod="openshift-multus/multus-scw67"
May 25 13:05:50.135600 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.135387    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/904a1760-12f0-49b4-a389-c71b39a99619-etc-systemd\") pod \"tuned-lcmng\" (UID: \"904a1760-12f0-49b4-a389-c71b39a99619\") " pod="openshift-cluster-node-tuning-operator/tuned-lcmng"
May 25 13:05:50.135712 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.135609    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/87942b76-e2a7-407e-9a99-2ea92f737157-multus-daemon-config\") pod \"multus-scw67\" (UID: \"87942b76-e2a7-407e-9a99-2ea92f737157\") " pod="openshift-multus/multus-scw67"
May 25 13:05:50.135810 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.135787    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/904a1760-12f0-49b4-a389-c71b39a99619-tmp\") pod \"tuned-lcmng\" (UID: \"904a1760-12f0-49b4-a389-c71b39a99619\") " pod="openshift-cluster-node-tuning-operator/tuned-lcmng"
May 25 13:05:50.136737 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.136714    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/55e3ea80-f133-45b9-b3b6-0d1abdbb1ad0-agent-certs\") pod \"konnectivity-agent-vwnkg\" (UID: \"55e3ea80-f133-45b9-b3b6-0d1abdbb1ad0\") " pod="kube-system/konnectivity-agent-vwnkg"
May 25 13:05:50.137555 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.137518    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/904a1760-12f0-49b4-a389-c71b39a99619-etc-tuned\") pod \"tuned-lcmng\" (UID: \"904a1760-12f0-49b4-a389-c71b39a99619\") " pod="openshift-cluster-node-tuning-operator/tuned-lcmng"
May 25 13:05:50.140561 ip-10-0-143-235 kubenswrapper[2557]: E0525 13:05:50.140540    2557 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 25 13:05:50.140657 ip-10-0-143-235 kubenswrapper[2557]: E0525 13:05:50.140566    2557 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 25 13:05:50.140657 ip-10-0-143-235 kubenswrapper[2557]: E0525 13:05:50.140581    2557 projected.go:194] Error preparing data for projected volume kube-api-access-4f7rk for pod openshift-network-diagnostics/network-check-target-96mr4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 13:05:50.140819 ip-10-0-143-235 kubenswrapper[2557]: E0525 13:05:50.140666    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a8d6074e-8880-4fd2-b612-de33be9d873e-kube-api-access-4f7rk podName:a8d6074e-8880-4fd2-b612-de33be9d873e nodeName:}" failed. No retries permitted until 2026-05-25 13:05:50.640648278 +0000 UTC m=+3.103664425 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-4f7rk" (UniqueName: "kubernetes.io/projected/a8d6074e-8880-4fd2-b612-de33be9d873e-kube-api-access-4f7rk") pod "network-check-target-96mr4" (UID: "a8d6074e-8880-4fd2-b612-de33be9d873e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 13:05:50.141670 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.141645    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bk5mv\" (UniqueName: \"kubernetes.io/projected/9d1a5d53-d761-478d-a85e-0974f65121e3-kube-api-access-bk5mv\") pod \"node-ca-v9wvs\" (UID: \"9d1a5d53-d761-478d-a85e-0974f65121e3\") " pod="openshift-image-registry/node-ca-v9wvs"
May 25 13:05:50.142578 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.142534    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pb9vv\" (UniqueName: \"kubernetes.io/projected/b3857de1-9f56-4063-b132-410d217c9f8b-kube-api-access-pb9vv\") pod \"aws-ebs-csi-driver-node-jc4n4\" (UID: \"b3857de1-9f56-4063-b132-410d217c9f8b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jc4n4"
May 25 13:05:50.143294 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.143257    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6f5ql\" (UniqueName: \"kubernetes.io/projected/71fb8f1c-14f2-441f-bbbf-92696b6559af-kube-api-access-6f5ql\") pod \"iptables-alerter-bwz4h\" (UID: \"71fb8f1c-14f2-441f-bbbf-92696b6559af\") " pod="openshift-network-operator/iptables-alerter-bwz4h"
May 25 13:05:50.143993 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.143639    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-96fp8\" (UniqueName: \"kubernetes.io/projected/d21bdf8b-d0ab-48f2-8736-54e92947dd53-kube-api-access-96fp8\") pod \"ovnkube-node-vqnvc\" (UID: \"d21bdf8b-d0ab-48f2-8736-54e92947dd53\") " pod="openshift-ovn-kubernetes/ovnkube-node-vqnvc"
May 25 13:05:50.143993 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.143944    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8r6b\" (UniqueName: \"kubernetes.io/projected/8c2f1987-8965-449c-b426-b46515332392-kube-api-access-j8r6b\") pod \"network-metrics-daemon-5lm4j\" (UID: \"8c2f1987-8965-449c-b426-b46515332392\") " pod="openshift-multus/network-metrics-daemon-5lm4j"
May 25 13:05:50.144129 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.144048    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9d77\" (UniqueName: \"kubernetes.io/projected/1dcac962-5867-4644-b3b3-ce62b3216b79-kube-api-access-w9d77\") pod \"multus-additional-cni-plugins-lphh6\" (UID: \"1dcac962-5867-4644-b3b3-ce62b3216b79\") " pod="openshift-multus/multus-additional-cni-plugins-lphh6"
May 25 13:05:50.144129 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.144074    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xx67l\" (UniqueName: \"kubernetes.io/projected/87942b76-e2a7-407e-9a99-2ea92f737157-kube-api-access-xx67l\") pod \"multus-scw67\" (UID: \"87942b76-e2a7-407e-9a99-2ea92f737157\") " pod="openshift-multus/multus-scw67"
May 25 13:05:50.145409 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.145388    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvzsd\" (UniqueName: \"kubernetes.io/projected/904a1760-12f0-49b4-a389-c71b39a99619-kube-api-access-kvzsd\") pod \"tuned-lcmng\" (UID: \"904a1760-12f0-49b4-a389-c71b39a99619\") " pod="openshift-cluster-node-tuning-operator/tuned-lcmng"
May 25 13:05:50.311768 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.311726    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-bwz4h"
May 25 13:05:50.318845 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.318818    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-lcmng"
May 25 13:05:50.326541 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.326513    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-v9wvs"
May 25 13:05:50.332212 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.332177    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-lphh6"
May 25 13:05:50.339675 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.339652    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-scw67"
May 25 13:05:50.346325 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.346304    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-vqnvc"
May 25 13:05:50.353007 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.352977    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-vwnkg"
May 25 13:05:50.358729 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.358702    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jc4n4"
May 25 13:05:50.431121 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.431089    2557 reflector.go:430] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
May 25 13:05:50.589174 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:50.589131    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1dcac962_5867_4644_b3b3_ce62b3216b79.slice/crio-ef5b91ec0236c6832e565535c1316559266dde1df0e97173d9f25b4954b79191 WatchSource:0}: Error finding container ef5b91ec0236c6832e565535c1316559266dde1df0e97173d9f25b4954b79191: Status 404 returned error can't find the container with id ef5b91ec0236c6832e565535c1316559266dde1df0e97173d9f25b4954b79191
May 25 13:05:50.590517 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:50.590494    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3857de1_9f56_4063_b132_410d217c9f8b.slice/crio-cfc8dafab2cdcaf1d9da3b3a2daa74e964c8329926c923d50e102c1f8cb7af25 WatchSource:0}: Error finding container cfc8dafab2cdcaf1d9da3b3a2daa74e964c8329926c923d50e102c1f8cb7af25: Status 404 returned error can't find the container with id cfc8dafab2cdcaf1d9da3b3a2daa74e964c8329926c923d50e102c1f8cb7af25
May 25 13:05:50.593613 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:50.593592    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55e3ea80_f133_45b9_b3b6_0d1abdbb1ad0.slice/crio-ae50568c09e85f5cd1b7c6916845d46584ff21bc376807c97109a6e6bed7b59a WatchSource:0}: Error finding container ae50568c09e85f5cd1b7c6916845d46584ff21bc376807c97109a6e6bed7b59a: Status 404 returned error can't find the container with id ae50568c09e85f5cd1b7c6916845d46584ff21bc376807c97109a6e6bed7b59a
May 25 13:05:50.594559 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:50.594537    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71fb8f1c_14f2_441f_bbbf_92696b6559af.slice/crio-a0c927334168a1d32d266111fa12cdb544c1bef1c458f4d9212c24dc2ef9865c WatchSource:0}: Error finding container a0c927334168a1d32d266111fa12cdb544c1bef1c458f4d9212c24dc2ef9865c: Status 404 returned error can't find the container with id a0c927334168a1d32d266111fa12cdb544c1bef1c458f4d9212c24dc2ef9865c
May 25 13:05:50.595839 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:50.595693    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd21bdf8b_d0ab_48f2_8736_54e92947dd53.slice/crio-d419c568f6a31f0531d8640e52024ef1b5485aed15a1dde3bbe254647b3896bb WatchSource:0}: Error finding container d419c568f6a31f0531d8640e52024ef1b5485aed15a1dde3bbe254647b3896bb: Status 404 returned error can't find the container with id d419c568f6a31f0531d8640e52024ef1b5485aed15a1dde3bbe254647b3896bb
May 25 13:05:50.597470 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:50.597207    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d1a5d53_d761_478d_a85e_0974f65121e3.slice/crio-b34f09f07fbbc8745ad5ec086112abff5c45915ce5d605314cd1967d31e1d134 WatchSource:0}: Error finding container b34f09f07fbbc8745ad5ec086112abff5c45915ce5d605314cd1967d31e1d134: Status 404 returned error can't find the container with id b34f09f07fbbc8745ad5ec086112abff5c45915ce5d605314cd1967d31e1d134
May 25 13:05:50.598728 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:50.597848    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87942b76_e2a7_407e_9a99_2ea92f737157.slice/crio-bf9a43182ed01b0535f694c82f4cbecc9c00a3ef77ab599295c4dd8bbb8ee9c3 WatchSource:0}: Error finding container bf9a43182ed01b0535f694c82f4cbecc9c00a3ef77ab599295c4dd8bbb8ee9c3: Status 404 returned error can't find the container with id bf9a43182ed01b0535f694c82f4cbecc9c00a3ef77ab599295c4dd8bbb8ee9c3
May 25 13:05:50.599645 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:05:50.599541    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod904a1760_12f0_49b4_a389_c71b39a99619.slice/crio-e4adecd56d2100aab7e41aeb1540b43dc6fac29932e1e0aff39df8244c93ce8d WatchSource:0}: Error finding container e4adecd56d2100aab7e41aeb1540b43dc6fac29932e1e0aff39df8244c93ce8d: Status 404 returned error can't find the container with id e4adecd56d2100aab7e41aeb1540b43dc6fac29932e1e0aff39df8244c93ce8d
May 25 13:05:50.638607 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.638578    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c2f1987-8965-449c-b426-b46515332392-metrics-certs\") pod \"network-metrics-daemon-5lm4j\" (UID: \"8c2f1987-8965-449c-b426-b46515332392\") " pod="openshift-multus/network-metrics-daemon-5lm4j"
May 25 13:05:50.638800 ip-10-0-143-235 kubenswrapper[2557]: E0525 13:05:50.638714    2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 13:05:50.638800 ip-10-0-143-235 kubenswrapper[2557]: E0525 13:05:50.638774    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c2f1987-8965-449c-b426-b46515332392-metrics-certs podName:8c2f1987-8965-449c-b426-b46515332392 nodeName:}" failed. No retries permitted until 2026-05-25 13:05:51.638759231 +0000 UTC m=+4.101775366 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8c2f1987-8965-449c-b426-b46515332392-metrics-certs") pod "network-metrics-daemon-5lm4j" (UID: "8c2f1987-8965-449c-b426-b46515332392") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 13:05:50.739380 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:50.739327    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-4f7rk\" (UniqueName: \"kubernetes.io/projected/a8d6074e-8880-4fd2-b612-de33be9d873e-kube-api-access-4f7rk\") pod \"network-check-target-96mr4\" (UID: \"a8d6074e-8880-4fd2-b612-de33be9d873e\") " pod="openshift-network-diagnostics/network-check-target-96mr4"
May 25 13:05:50.739557 ip-10-0-143-235 kubenswrapper[2557]: E0525 13:05:50.739501    2557 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 25 13:05:50.739557 ip-10-0-143-235 kubenswrapper[2557]: E0525 13:05:50.739524    2557 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 25 13:05:50.739557 ip-10-0-143-235 kubenswrapper[2557]: E0525 13:05:50.739536    2557 projected.go:194] Error preparing data for projected volume kube-api-access-4f7rk for pod openshift-network-diagnostics/network-check-target-96mr4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 13:05:50.739695 ip-10-0-143-235 kubenswrapper[2557]: E0525 13:05:50.739598    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a8d6074e-8880-4fd2-b612-de33be9d873e-kube-api-access-4f7rk podName:a8d6074e-8880-4fd2-b612-de33be9d873e nodeName:}" failed. No retries permitted until 2026-05-25 13:05:51.739578236 +0000 UTC m=+4.202594389 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-4f7rk" (UniqueName: "kubernetes.io/projected/a8d6074e-8880-4fd2-b612-de33be9d873e-kube-api-access-4f7rk") pod "network-check-target-96mr4" (UID: "a8d6074e-8880-4fd2-b612-de33be9d873e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 13:05:51.060860 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:51.060813    2557 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-05-24 13:00:49 +0000 UTC" deadline="2027-11-19 22:24:51.753718654 +0000 UTC"
May 25 13:05:51.060860 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:51.060856    2557 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13041h19m0.692866218s"
May 25 13:05:51.146178 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:51.146098    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-96mr4"
May 25 13:05:51.146384 ip-10-0-143-235 kubenswrapper[2557]: E0525 13:05:51.146239    2557 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-96mr4" podUID="a8d6074e-8880-4fd2-b612-de33be9d873e"
May 25 13:05:51.164393 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:51.164357    2557 generic.go:358] "Generic (PLEG): container finished" podID="f10bc5abc6a23ce724748b3d53d4900b" containerID="534e8a512ba101359fa823e410f56b3447aefc2da6130bb3dbc8cc9c1cec03d1" exitCode=0
May 25 13:05:51.164554 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:51.164458    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-235.ec2.internal" event={"ID":"f10bc5abc6a23ce724748b3d53d4900b","Type":"ContainerDied","Data":"534e8a512ba101359fa823e410f56b3447aefc2da6130bb3dbc8cc9c1cec03d1"}
May 25 13:05:51.194103 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:51.194063    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-scw67" event={"ID":"87942b76-e2a7-407e-9a99-2ea92f737157","Type":"ContainerStarted","Data":"bf9a43182ed01b0535f694c82f4cbecc9c00a3ef77ab599295c4dd8bbb8ee9c3"}
May 25 13:05:51.205049 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:51.204983    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-bwz4h" event={"ID":"71fb8f1c-14f2-441f-bbbf-92696b6559af","Type":"ContainerStarted","Data":"a0c927334168a1d32d266111fa12cdb544c1bef1c458f4d9212c24dc2ef9865c"}
May 25 13:05:51.209295 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:51.209265    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-235.ec2.internal" event={"ID":"be380709ced6c0e0087779b84acc1278","Type":"ContainerStarted","Data":"38251efe763a38dc33221ddd93539baf4758b5563e7671fa4538e4a2d93381f6"}
May 25 13:05:51.217179 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:51.217131    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-lcmng" event={"ID":"904a1760-12f0-49b4-a389-c71b39a99619","Type":"ContainerStarted","Data":"e4adecd56d2100aab7e41aeb1540b43dc6fac29932e1e0aff39df8244c93ce8d"}
May 25 13:05:51.237105 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:51.237016    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-v9wvs" event={"ID":"9d1a5d53-d761-478d-a85e-0974f65121e3","Type":"ContainerStarted","Data":"b34f09f07fbbc8745ad5ec086112abff5c45915ce5d605314cd1967d31e1d134"}
May 25 13:05:51.251740 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:51.251654    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vqnvc" event={"ID":"d21bdf8b-d0ab-48f2-8736-54e92947dd53","Type":"ContainerStarted","Data":"d419c568f6a31f0531d8640e52024ef1b5485aed15a1dde3bbe254647b3896bb"}
May 25 13:05:51.255540 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:51.255234    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-vwnkg" event={"ID":"55e3ea80-f133-45b9-b3b6-0d1abdbb1ad0","Type":"ContainerStarted","Data":"ae50568c09e85f5cd1b7c6916845d46584ff21bc376807c97109a6e6bed7b59a"}
May 25 13:05:51.260440 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:51.260406    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jc4n4" event={"ID":"b3857de1-9f56-4063-b132-410d217c9f8b","Type":"ContainerStarted","Data":"cfc8dafab2cdcaf1d9da3b3a2daa74e964c8329926c923d50e102c1f8cb7af25"}
May 25 13:05:51.274940 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:51.274900    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lphh6" event={"ID":"1dcac962-5867-4644-b3b3-ce62b3216b79","Type":"ContainerStarted","Data":"ef5b91ec0236c6832e565535c1316559266dde1df0e97173d9f25b4954b79191"}
May 25 13:05:51.593076 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:51.593043    2557 reflector.go:430] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
May 25 13:05:51.647539 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:51.646915    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c2f1987-8965-449c-b426-b46515332392-metrics-certs\") pod \"network-metrics-daemon-5lm4j\" (UID: \"8c2f1987-8965-449c-b426-b46515332392\") " pod="openshift-multus/network-metrics-daemon-5lm4j"
May 25 13:05:51.647539 ip-10-0-143-235 kubenswrapper[2557]: E0525 13:05:51.647072    2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 13:05:51.647539 ip-10-0-143-235 kubenswrapper[2557]: E0525 13:05:51.647137    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c2f1987-8965-449c-b426-b46515332392-metrics-certs podName:8c2f1987-8965-449c-b426-b46515332392 nodeName:}" failed. No retries permitted until 2026-05-25 13:05:53.647117436 +0000 UTC m=+6.110133594 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8c2f1987-8965-449c-b426-b46515332392-metrics-certs") pod "network-metrics-daemon-5lm4j" (UID: "8c2f1987-8965-449c-b426-b46515332392") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 13:05:51.748004 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:51.747905    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-4f7rk\" (UniqueName: \"kubernetes.io/projected/a8d6074e-8880-4fd2-b612-de33be9d873e-kube-api-access-4f7rk\") pod \"network-check-target-96mr4\" (UID: \"a8d6074e-8880-4fd2-b612-de33be9d873e\") " pod="openshift-network-diagnostics/network-check-target-96mr4"
May 25 13:05:51.748176 ip-10-0-143-235 kubenswrapper[2557]: E0525 13:05:51.748077    2557 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 25 13:05:51.748176 ip-10-0-143-235 kubenswrapper[2557]: E0525 13:05:51.748095    2557 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 25 13:05:51.748176 ip-10-0-143-235 kubenswrapper[2557]: E0525 13:05:51.748107    2557 projected.go:194] Error preparing data for projected volume kube-api-access-4f7rk for pod openshift-network-diagnostics/network-check-target-96mr4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 13:05:51.748176 ip-10-0-143-235 kubenswrapper[2557]: E0525 13:05:51.748164    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a8d6074e-8880-4fd2-b612-de33be9d873e-kube-api-access-4f7rk podName:a8d6074e-8880-4fd2-b612-de33be9d873e nodeName:}" failed. No retries permitted until 2026-05-25 13:05:53.748147252 +0000 UTC m=+6.211163394 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-4f7rk" (UniqueName: "kubernetes.io/projected/a8d6074e-8880-4fd2-b612-de33be9d873e-kube-api-access-4f7rk") pod "network-check-target-96mr4" (UID: "a8d6074e-8880-4fd2-b612-de33be9d873e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 13:05:52.147927 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:52.147858    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5lm4j"
May 25 13:05:52.148389 ip-10-0-143-235 kubenswrapper[2557]: E0525 13:05:52.147998    2557 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5lm4j" podUID="8c2f1987-8965-449c-b426-b46515332392"
May 25 13:05:52.289456 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:52.289421    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-235.ec2.internal" event={"ID":"f10bc5abc6a23ce724748b3d53d4900b","Type":"ContainerStarted","Data":"55d37cde4501548ffb3e1f10cc5f531ab98155084a0206c93f1109e8a7e9ae1d"}
May 25 13:05:52.303089 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:52.302509    2557 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-235.ec2.internal" podStartSLOduration=3.302489242 podStartE2EDuration="3.302489242s" podCreationTimestamp="2026-05-25 13:05:49 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 13:05:51.222607981 +0000 UTC m=+3.685624142" watchObservedRunningTime="2026-05-25 13:05:52.302489242 +0000 UTC m=+4.765505400"
May 25 13:05:53.145584 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:53.145549    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-96mr4"
May 25 13:05:53.145764 ip-10-0-143-235 kubenswrapper[2557]: E0525 13:05:53.145686    2557 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-96mr4" podUID="a8d6074e-8880-4fd2-b612-de33be9d873e"
May 25 13:05:53.663411 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:53.663358    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c2f1987-8965-449c-b426-b46515332392-metrics-certs\") pod \"network-metrics-daemon-5lm4j\" (UID: \"8c2f1987-8965-449c-b426-b46515332392\") " pod="openshift-multus/network-metrics-daemon-5lm4j"
May 25 13:05:53.663861 ip-10-0-143-235 kubenswrapper[2557]: E0525 13:05:53.663488    2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 13:05:53.663861 ip-10-0-143-235 kubenswrapper[2557]: E0525 13:05:53.663577    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c2f1987-8965-449c-b426-b46515332392-metrics-certs podName:8c2f1987-8965-449c-b426-b46515332392 nodeName:}" failed. No retries permitted until 2026-05-25 13:05:57.663554911 +0000 UTC m=+10.126571053 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8c2f1987-8965-449c-b426-b46515332392-metrics-certs") pod "network-metrics-daemon-5lm4j" (UID: "8c2f1987-8965-449c-b426-b46515332392") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 13:05:53.764724 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:53.764681    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-4f7rk\" (UniqueName: \"kubernetes.io/projected/a8d6074e-8880-4fd2-b612-de33be9d873e-kube-api-access-4f7rk\") pod \"network-check-target-96mr4\" (UID: \"a8d6074e-8880-4fd2-b612-de33be9d873e\") " pod="openshift-network-diagnostics/network-check-target-96mr4"
May 25 13:05:53.764944 ip-10-0-143-235 kubenswrapper[2557]: E0525 13:05:53.764892    2557 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 25 13:05:53.764944 ip-10-0-143-235 kubenswrapper[2557]: E0525 13:05:53.764920    2557 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 25 13:05:53.764944 ip-10-0-143-235 kubenswrapper[2557]: E0525 13:05:53.764934    2557 projected.go:194] Error preparing data for projected volume kube-api-access-4f7rk for pod openshift-network-diagnostics/network-check-target-96mr4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 13:05:53.765078 ip-10-0-143-235 kubenswrapper[2557]: E0525 13:05:53.765003    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a8d6074e-8880-4fd2-b612-de33be9d873e-kube-api-access-4f7rk podName:a8d6074e-8880-4fd2-b612-de33be9d873e nodeName:}" failed. No retries permitted until 2026-05-25 13:05:57.764984473 +0000 UTC m=+10.228000613 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-4f7rk" (UniqueName: "kubernetes.io/projected/a8d6074e-8880-4fd2-b612-de33be9d873e-kube-api-access-4f7rk") pod "network-check-target-96mr4" (UID: "a8d6074e-8880-4fd2-b612-de33be9d873e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 13:05:54.146765 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:54.146193    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5lm4j"
May 25 13:05:54.146765 ip-10-0-143-235 kubenswrapper[2557]: E0525 13:05:54.146337    2557 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5lm4j" podUID="8c2f1987-8965-449c-b426-b46515332392"
May 25 13:05:55.145922 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:55.145769    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-96mr4"
May 25 13:05:55.146413 ip-10-0-143-235 kubenswrapper[2557]: E0525 13:05:55.145926    2557 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-96mr4" podUID="a8d6074e-8880-4fd2-b612-de33be9d873e"
May 25 13:05:56.146032 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:56.145666    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5lm4j"
May 25 13:05:56.146032 ip-10-0-143-235 kubenswrapper[2557]: E0525 13:05:56.145812    2557 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5lm4j" podUID="8c2f1987-8965-449c-b426-b46515332392"
May 25 13:05:57.146013 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:57.145970    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-96mr4"
May 25 13:05:57.146175 ip-10-0-143-235 kubenswrapper[2557]: E0525 13:05:57.146106    2557 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-96mr4" podUID="a8d6074e-8880-4fd2-b612-de33be9d873e"
May 25 13:05:57.694706 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:57.694600    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c2f1987-8965-449c-b426-b46515332392-metrics-certs\") pod \"network-metrics-daemon-5lm4j\" (UID: \"8c2f1987-8965-449c-b426-b46515332392\") " pod="openshift-multus/network-metrics-daemon-5lm4j"
May 25 13:05:57.694879 ip-10-0-143-235 kubenswrapper[2557]: E0525 13:05:57.694722    2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 13:05:57.694879 ip-10-0-143-235 kubenswrapper[2557]: E0525 13:05:57.694786    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c2f1987-8965-449c-b426-b46515332392-metrics-certs podName:8c2f1987-8965-449c-b426-b46515332392 nodeName:}" failed. No retries permitted until 2026-05-25 13:06:05.694767989 +0000 UTC m=+18.157784129 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8c2f1987-8965-449c-b426-b46515332392-metrics-certs") pod "network-metrics-daemon-5lm4j" (UID: "8c2f1987-8965-449c-b426-b46515332392") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 13:05:57.795953 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:57.795816    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-4f7rk\" (UniqueName: \"kubernetes.io/projected/a8d6074e-8880-4fd2-b612-de33be9d873e-kube-api-access-4f7rk\") pod \"network-check-target-96mr4\" (UID: \"a8d6074e-8880-4fd2-b612-de33be9d873e\") " pod="openshift-network-diagnostics/network-check-target-96mr4"
May 25 13:05:57.796151 ip-10-0-143-235 kubenswrapper[2557]: E0525 13:05:57.796030    2557 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 25 13:05:57.796151 ip-10-0-143-235 kubenswrapper[2557]: E0525 13:05:57.796056    2557 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 25 13:05:57.796151 ip-10-0-143-235 kubenswrapper[2557]: E0525 13:05:57.796070    2557 projected.go:194] Error preparing data for projected volume kube-api-access-4f7rk for pod openshift-network-diagnostics/network-check-target-96mr4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 13:05:57.796151 ip-10-0-143-235 kubenswrapper[2557]: E0525 13:05:57.796139    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a8d6074e-8880-4fd2-b612-de33be9d873e-kube-api-access-4f7rk podName:a8d6074e-8880-4fd2-b612-de33be9d873e nodeName:}" failed. No retries permitted until 2026-05-25 13:06:05.796118715 +0000 UTC m=+18.259134864 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-4f7rk" (UniqueName: "kubernetes.io/projected/a8d6074e-8880-4fd2-b612-de33be9d873e-kube-api-access-4f7rk") pod "network-check-target-96mr4" (UID: "a8d6074e-8880-4fd2-b612-de33be9d873e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 13:05:58.147118 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:58.147083    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5lm4j"
May 25 13:05:58.147597 ip-10-0-143-235 kubenswrapper[2557]: E0525 13:05:58.147223    2557 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5lm4j" podUID="8c2f1987-8965-449c-b426-b46515332392"
May 25 13:05:59.145396 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:05:59.145361    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-96mr4"
May 25 13:05:59.145640 ip-10-0-143-235 kubenswrapper[2557]: E0525 13:05:59.145491    2557 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-96mr4" podUID="a8d6074e-8880-4fd2-b612-de33be9d873e"
May 25 13:06:00.148433 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:00.148397    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5lm4j"
May 25 13:06:00.148860 ip-10-0-143-235 kubenswrapper[2557]: E0525 13:06:00.148517    2557 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5lm4j" podUID="8c2f1987-8965-449c-b426-b46515332392"
May 25 13:06:01.145629 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:01.145588    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-96mr4"
May 25 13:06:01.145825 ip-10-0-143-235 kubenswrapper[2557]: E0525 13:06:01.145704    2557 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-96mr4" podUID="a8d6074e-8880-4fd2-b612-de33be9d873e"
May 25 13:06:02.146191 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:02.146155    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5lm4j"
May 25 13:06:02.146642 ip-10-0-143-235 kubenswrapper[2557]: E0525 13:06:02.146310    2557 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5lm4j" podUID="8c2f1987-8965-449c-b426-b46515332392"
May 25 13:06:03.145364 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:03.145317    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-96mr4"
May 25 13:06:03.145560 ip-10-0-143-235 kubenswrapper[2557]: E0525 13:06:03.145443    2557 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-96mr4" podUID="a8d6074e-8880-4fd2-b612-de33be9d873e"
May 25 13:06:04.146083 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:04.146042    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5lm4j"
May 25 13:06:04.146555 ip-10-0-143-235 kubenswrapper[2557]: E0525 13:06:04.146193    2557 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5lm4j" podUID="8c2f1987-8965-449c-b426-b46515332392"
May 25 13:06:05.146169 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:05.146096    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-96mr4"
May 25 13:06:05.146687 ip-10-0-143-235 kubenswrapper[2557]: E0525 13:06:05.146228    2557 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-96mr4" podUID="a8d6074e-8880-4fd2-b612-de33be9d873e"
May 25 13:06:05.756590 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:05.756555    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c2f1987-8965-449c-b426-b46515332392-metrics-certs\") pod \"network-metrics-daemon-5lm4j\" (UID: \"8c2f1987-8965-449c-b426-b46515332392\") " pod="openshift-multus/network-metrics-daemon-5lm4j"
May 25 13:06:05.756765 ip-10-0-143-235 kubenswrapper[2557]: E0525 13:06:05.756676    2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 13:06:05.756765 ip-10-0-143-235 kubenswrapper[2557]: E0525 13:06:05.756737    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c2f1987-8965-449c-b426-b46515332392-metrics-certs podName:8c2f1987-8965-449c-b426-b46515332392 nodeName:}" failed. No retries permitted until 2026-05-25 13:06:21.756719184 +0000 UTC m=+34.219735323 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8c2f1987-8965-449c-b426-b46515332392-metrics-certs") pod "network-metrics-daemon-5lm4j" (UID: "8c2f1987-8965-449c-b426-b46515332392") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 13:06:05.857665 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:05.857627    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-4f7rk\" (UniqueName: \"kubernetes.io/projected/a8d6074e-8880-4fd2-b612-de33be9d873e-kube-api-access-4f7rk\") pod \"network-check-target-96mr4\" (UID: \"a8d6074e-8880-4fd2-b612-de33be9d873e\") " pod="openshift-network-diagnostics/network-check-target-96mr4"
May 25 13:06:05.857858 ip-10-0-143-235 kubenswrapper[2557]: E0525 13:06:05.857837    2557 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 25 13:06:05.857948 ip-10-0-143-235 kubenswrapper[2557]: E0525 13:06:05.857866    2557 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 25 13:06:05.857948 ip-10-0-143-235 kubenswrapper[2557]: E0525 13:06:05.857881    2557 projected.go:194] Error preparing data for projected volume kube-api-access-4f7rk for pod openshift-network-diagnostics/network-check-target-96mr4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 13:06:05.858045 ip-10-0-143-235 kubenswrapper[2557]: E0525 13:06:05.857949    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a8d6074e-8880-4fd2-b612-de33be9d873e-kube-api-access-4f7rk podName:a8d6074e-8880-4fd2-b612-de33be9d873e nodeName:}" failed. No retries permitted until 2026-05-25 13:06:21.857929068 +0000 UTC m=+34.320945205 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-4f7rk" (UniqueName: "kubernetes.io/projected/a8d6074e-8880-4fd2-b612-de33be9d873e-kube-api-access-4f7rk") pod "network-check-target-96mr4" (UID: "a8d6074e-8880-4fd2-b612-de33be9d873e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 13:06:06.145501 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:06.145469    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5lm4j"
May 25 13:06:06.145684 ip-10-0-143-235 kubenswrapper[2557]: E0525 13:06:06.145616    2557 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5lm4j" podUID="8c2f1987-8965-449c-b426-b46515332392"
May 25 13:06:07.042341 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:07.042291    2557 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-235.ec2.internal" podStartSLOduration=18.042275259 podStartE2EDuration="18.042275259s" podCreationTimestamp="2026-05-25 13:05:49 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 13:05:52.30298919 +0000 UTC m=+4.766005348" watchObservedRunningTime="2026-05-25 13:06:07.042275259 +0000 UTC m=+19.505291416"
May 25 13:06:07.042864 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:07.042840    2557 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-hp2w9"]
May 25 13:06:07.073311 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:07.073272    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-hp2w9"
May 25 13:06:07.075902 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:07.075867    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
May 25 13:06:07.076056 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:07.076010    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
May 25 13:06:07.076294 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:07.076274    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-wmpmm\""
May 25 13:06:07.145803 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:07.145769    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-96mr4"
May 25 13:06:07.146002 ip-10-0-143-235 kubenswrapper[2557]: E0525 13:06:07.145926    2557 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-96mr4" podUID="a8d6074e-8880-4fd2-b612-de33be9d873e"
May 25 13:06:07.169119 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:07.169078    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/50653f82-1790-4f22-94c1-406308112ca5-tmp-dir\") pod \"node-resolver-hp2w9\" (UID: \"50653f82-1790-4f22-94c1-406308112ca5\") " pod="openshift-dns/node-resolver-hp2w9"
May 25 13:06:07.169119 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:07.169122    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/50653f82-1790-4f22-94c1-406308112ca5-hosts-file\") pod \"node-resolver-hp2w9\" (UID: \"50653f82-1790-4f22-94c1-406308112ca5\") " pod="openshift-dns/node-resolver-hp2w9"
May 25 13:06:07.169443 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:07.169236    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-295dm\" (UniqueName: \"kubernetes.io/projected/50653f82-1790-4f22-94c1-406308112ca5-kube-api-access-295dm\") pod \"node-resolver-hp2w9\" (UID: \"50653f82-1790-4f22-94c1-406308112ca5\") " pod="openshift-dns/node-resolver-hp2w9"
May 25 13:06:07.269909 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:07.269865    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-295dm\" (UniqueName: \"kubernetes.io/projected/50653f82-1790-4f22-94c1-406308112ca5-kube-api-access-295dm\") pod \"node-resolver-hp2w9\" (UID: \"50653f82-1790-4f22-94c1-406308112ca5\") " pod="openshift-dns/node-resolver-hp2w9"
May 25 13:06:07.270095 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:07.269956    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/50653f82-1790-4f22-94c1-406308112ca5-tmp-dir\") pod \"node-resolver-hp2w9\" (UID: \"50653f82-1790-4f22-94c1-406308112ca5\") " pod="openshift-dns/node-resolver-hp2w9"
May 25 13:06:07.270095 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:07.269987    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/50653f82-1790-4f22-94c1-406308112ca5-hosts-file\") pod \"node-resolver-hp2w9\" (UID: \"50653f82-1790-4f22-94c1-406308112ca5\") " pod="openshift-dns/node-resolver-hp2w9"
May 25 13:06:07.270095 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:07.270074    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/50653f82-1790-4f22-94c1-406308112ca5-hosts-file\") pod \"node-resolver-hp2w9\" (UID: \"50653f82-1790-4f22-94c1-406308112ca5\") " pod="openshift-dns/node-resolver-hp2w9"
May 25 13:06:07.270361 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:07.270322    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/50653f82-1790-4f22-94c1-406308112ca5-tmp-dir\") pod \"node-resolver-hp2w9\" (UID: \"50653f82-1790-4f22-94c1-406308112ca5\") " pod="openshift-dns/node-resolver-hp2w9"
May 25 13:06:07.279752 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:07.279725    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-295dm\" (UniqueName: \"kubernetes.io/projected/50653f82-1790-4f22-94c1-406308112ca5-kube-api-access-295dm\") pod \"node-resolver-hp2w9\" (UID: \"50653f82-1790-4f22-94c1-406308112ca5\") " pod="openshift-dns/node-resolver-hp2w9"
May 25 13:06:07.383132 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:07.383101    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-hp2w9"
May 25 13:06:08.146293 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:08.146260    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5lm4j"
May 25 13:06:08.146666 ip-10-0-143-235 kubenswrapper[2557]: E0525 13:06:08.146371    2557 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5lm4j" podUID="8c2f1987-8965-449c-b426-b46515332392"
May 25 13:06:08.545050 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:06:08.545024    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50653f82_1790_4f22_94c1_406308112ca5.slice/crio-95570ffb8364aa488aa59dddff568e2eae01d2f3a895b43e6dd3c479536d6b17 WatchSource:0}: Error finding container 95570ffb8364aa488aa59dddff568e2eae01d2f3a895b43e6dd3c479536d6b17: Status 404 returned error can't find the container with id 95570ffb8364aa488aa59dddff568e2eae01d2f3a895b43e6dd3c479536d6b17
May 25 13:06:09.146148 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:09.145883    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-96mr4"
May 25 13:06:09.146208 ip-10-0-143-235 kubenswrapper[2557]: E0525 13:06:09.146160    2557 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-96mr4" podUID="a8d6074e-8880-4fd2-b612-de33be9d873e"
May 25 13:06:09.321635 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:09.321331    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-vwnkg" event={"ID":"55e3ea80-f133-45b9-b3b6-0d1abdbb1ad0","Type":"ContainerStarted","Data":"75f6912ef0b8c0ff89059ea6fffe8c41251b136954349d2acd6924d3423cb9ca"}
May 25 13:06:09.322900 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:09.322868    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jc4n4" event={"ID":"b3857de1-9f56-4063-b132-410d217c9f8b","Type":"ContainerStarted","Data":"e4b683140d6184a0e9b296819876079a402df71f734c19686a6f8efa8576a704"}
May 25 13:06:09.324170 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:09.324146    2557 generic.go:358] "Generic (PLEG): container finished" podID="1dcac962-5867-4644-b3b3-ce62b3216b79" containerID="e5ee65a65f8b3bec2621f54c76c386714032e94ff1ee54fab4350ebfcb1268e0" exitCode=0
May 25 13:06:09.324291 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:09.324215    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lphh6" event={"ID":"1dcac962-5867-4644-b3b3-ce62b3216b79","Type":"ContainerDied","Data":"e5ee65a65f8b3bec2621f54c76c386714032e94ff1ee54fab4350ebfcb1268e0"}
May 25 13:06:09.325801 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:09.325747    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-scw67" event={"ID":"87942b76-e2a7-407e-9a99-2ea92f737157","Type":"ContainerStarted","Data":"ecd9a4e5c3bc6ace2b12fd98833d71e990d71a4ee7fc3f461a193157639e7e80"}
May 25 13:06:09.327402 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:09.327376    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-hp2w9" event={"ID":"50653f82-1790-4f22-94c1-406308112ca5","Type":"ContainerStarted","Data":"3249609461b418130a66e745f735514f2c994e1ac2ebcef525d7f2b5d03e5b7a"}
May 25 13:06:09.327495 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:09.327414    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-hp2w9" event={"ID":"50653f82-1790-4f22-94c1-406308112ca5","Type":"ContainerStarted","Data":"95570ffb8364aa488aa59dddff568e2eae01d2f3a895b43e6dd3c479536d6b17"}
May 25 13:06:09.328930 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:09.328903    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-lcmng" event={"ID":"904a1760-12f0-49b4-a389-c71b39a99619","Type":"ContainerStarted","Data":"85f34fd7813784202e80d23712b17dd005654e2cf076a4e1906598097125bc18"}
May 25 13:06:09.330240 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:09.330216    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-v9wvs" event={"ID":"9d1a5d53-d761-478d-a85e-0974f65121e3","Type":"ContainerStarted","Data":"429ed8d053a10761df8dbc062e3300e2a12395f3e9aa6149852f2e44d98feceb"}
May 25 13:06:09.332552 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:09.332529    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vqnvc_d21bdf8b-d0ab-48f2-8736-54e92947dd53/ovn-acl-logging/0.log"
May 25 13:06:09.332888 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:09.332865    2557 generic.go:358] "Generic (PLEG): container finished" podID="d21bdf8b-d0ab-48f2-8736-54e92947dd53" containerID="e97710c4274bdc36d4fa397c0a9bc04b0d71dc6365a4a445877d223957e22c0d" exitCode=1
May 25 13:06:09.332961 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:09.332901    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vqnvc" event={"ID":"d21bdf8b-d0ab-48f2-8736-54e92947dd53","Type":"ContainerStarted","Data":"f3933c034f1c13fc538b6a6a8a3d6f2d5f377ec479520d6e4723406f030ad12d"}
May 25 13:06:09.332961 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:09.332919    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vqnvc" event={"ID":"d21bdf8b-d0ab-48f2-8736-54e92947dd53","Type":"ContainerStarted","Data":"db7f241bebffc115fc8d63f5afb86577f186536f08f27d49b89de3388e2c93ef"}
May 25 13:06:09.332961 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:09.332930    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vqnvc" event={"ID":"d21bdf8b-d0ab-48f2-8736-54e92947dd53","Type":"ContainerDied","Data":"e97710c4274bdc36d4fa397c0a9bc04b0d71dc6365a4a445877d223957e22c0d"}
May 25 13:06:09.332961 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:09.332940    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vqnvc" event={"ID":"d21bdf8b-d0ab-48f2-8736-54e92947dd53","Type":"ContainerStarted","Data":"5cbc578bdbff8e15da8b75aebc782ed0d295a8c66deee8593686425e552b19d3"}
May 25 13:06:09.338769 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:09.338723    2557 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/konnectivity-agent-vwnkg" podStartSLOduration=3.417625881 podStartE2EDuration="21.33870681s" podCreationTimestamp="2026-05-25 13:05:48 +0000 UTC" firstStartedPulling="2026-05-25 13:05:50.595426128 +0000 UTC m=+3.058442263" lastFinishedPulling="2026-05-25 13:06:08.516507048 +0000 UTC m=+20.979523192" observedRunningTime="2026-05-25 13:06:09.337564734 +0000 UTC m=+21.800580891" watchObservedRunningTime="2026-05-25 13:06:09.33870681 +0000 UTC m=+21.801722968"
May 25 13:06:09.354060 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:09.354004    2557 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-lcmng" podStartSLOduration=3.412773174 podStartE2EDuration="21.35399051s" podCreationTimestamp="2026-05-25 13:05:48 +0000 UTC" firstStartedPulling="2026-05-25 13:05:50.601384581 +0000 UTC m=+3.064400731" lastFinishedPulling="2026-05-25 13:06:08.542601932 +0000 UTC m=+21.005618067" observedRunningTime="2026-05-25 13:06:09.353684454 +0000 UTC m=+21.816700614" watchObservedRunningTime="2026-05-25 13:06:09.35399051 +0000 UTC m=+21.817006665"
May 25 13:06:09.391708 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:09.391665    2557 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-dns/node-resolver-hp2w9" podStartSLOduration=2.391650108 podStartE2EDuration="2.391650108s" podCreationTimestamp="2026-05-25 13:06:07 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 13:06:09.391632608 +0000 UTC m=+21.854648765" watchObservedRunningTime="2026-05-25 13:06:09.391650108 +0000 UTC m=+21.854666265"
May 25 13:06:09.424562 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:09.424513    2557 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-image-registry/node-ca-v9wvs" podStartSLOduration=8.388696968 podStartE2EDuration="21.424496748s" podCreationTimestamp="2026-05-25 13:05:48 +0000 UTC" firstStartedPulling="2026-05-25 13:05:50.599246819 +0000 UTC m=+3.062262954" lastFinishedPulling="2026-05-25 13:06:03.635046589 +0000 UTC m=+16.098062734" observedRunningTime="2026-05-25 13:06:09.406414894 +0000 UTC m=+21.869431051" watchObservedRunningTime="2026-05-25 13:06:09.424496748 +0000 UTC m=+21.887512967"
May 25 13:06:09.424745 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:09.424679    2557 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/multus-scw67" podStartSLOduration=3.476513664 podStartE2EDuration="21.424674511s" podCreationTimestamp="2026-05-25 13:05:48 +0000 UTC" firstStartedPulling="2026-05-25 13:05:50.60007866 +0000 UTC m=+3.063094795" lastFinishedPulling="2026-05-25 13:06:08.548239507 +0000 UTC m=+21.011255642" observedRunningTime="2026-05-25 13:06:09.424373449 +0000 UTC m=+21.887389603" watchObservedRunningTime="2026-05-25 13:06:09.424674511 +0000 UTC m=+21.887690669"
May 25 13:06:09.862976 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:09.862936    2557 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
May 25 13:06:10.091459 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:10.091311    2557 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-05-25T13:06:09.862973123Z","UUID":"9c3beaa8-e0c7-41ca-b1ce-d8a3e3863253","Handler":null,"Name":"","Endpoint":""}
May 25 13:06:10.093698 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:10.093674    2557 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
May 25 13:06:10.093862 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:10.093716    2557 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
May 25 13:06:10.145720 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:10.145682    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5lm4j"
May 25 13:06:10.145900 ip-10-0-143-235 kubenswrapper[2557]: E0525 13:06:10.145820    2557 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5lm4j" podUID="8c2f1987-8965-449c-b426-b46515332392"
May 25 13:06:10.337162 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:10.336999    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-bwz4h" event={"ID":"71fb8f1c-14f2-441f-bbbf-92696b6559af","Type":"ContainerStarted","Data":"04f824ff6cf19c86e958b1c38b4f570ca6a03356bba5d8367e52f71f81e0ce8b"}
May 25 13:06:10.340933 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:10.340901    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vqnvc_d21bdf8b-d0ab-48f2-8736-54e92947dd53/ovn-acl-logging/0.log"
May 25 13:06:10.341374 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:10.341327    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vqnvc" event={"ID":"d21bdf8b-d0ab-48f2-8736-54e92947dd53","Type":"ContainerStarted","Data":"97c5dc46206cffa057d89fa6fa5a0d082c7b07d8c6f5c276b9ba8ffe53064622"}
May 25 13:06:10.341569 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:10.341382    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vqnvc" event={"ID":"d21bdf8b-d0ab-48f2-8736-54e92947dd53","Type":"ContainerStarted","Data":"7d6aba0275adb390ba45cc7ccc19e30326b85fa43f29a7b8f720244e35a3046f"}
May 25 13:06:10.343743 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:10.343715    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jc4n4" event={"ID":"b3857de1-9f56-4063-b132-410d217c9f8b","Type":"ContainerStarted","Data":"5daaa0a5d070fa1f3c1692e85cb9f8aa648b42f66242050e11d4e42e7199c696"}
May 25 13:06:11.145901 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:11.145691    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-96mr4"
May 25 13:06:11.146037 ip-10-0-143-235 kubenswrapper[2557]: E0525 13:06:11.146009    2557 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-96mr4" podUID="a8d6074e-8880-4fd2-b612-de33be9d873e"
May 25 13:06:11.347644 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:11.347526    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jc4n4" event={"ID":"b3857de1-9f56-4063-b132-410d217c9f8b","Type":"ContainerStarted","Data":"bb346f7d107ad5e6f4e9139c4e20b303e404b983c42e23e12ae9319c42b88b45"}
May 25 13:06:11.367258 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:11.367196    2557 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jc4n4" podStartSLOduration=2.9723876860000003 podStartE2EDuration="23.367173984s" podCreationTimestamp="2026-05-25 13:05:48 +0000 UTC" firstStartedPulling="2026-05-25 13:05:50.592519714 +0000 UTC m=+3.055535849" lastFinishedPulling="2026-05-25 13:06:10.987305967 +0000 UTC m=+23.450322147" observedRunningTime="2026-05-25 13:06:11.366316563 +0000 UTC m=+23.829332720" watchObservedRunningTime="2026-05-25 13:06:11.367173984 +0000 UTC m=+23.830190142"
May 25 13:06:11.367454 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:11.367427    2557 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-bwz4h" podStartSLOduration=5.447472352 podStartE2EDuration="23.367417819s" podCreationTimestamp="2026-05-25 13:05:48 +0000 UTC" firstStartedPulling="2026-05-25 13:05:50.596559398 +0000 UTC m=+3.059575551" lastFinishedPulling="2026-05-25 13:06:08.516504864 +0000 UTC m=+20.979521018" observedRunningTime="2026-05-25 13:06:10.355419537 +0000 UTC m=+22.818435695" watchObservedRunningTime="2026-05-25 13:06:11.367417819 +0000 UTC m=+23.830433981"
May 25 13:06:12.145900 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:12.145863    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5lm4j"
May 25 13:06:12.146087 ip-10-0-143-235 kubenswrapper[2557]: E0525 13:06:12.145998    2557 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5lm4j" podUID="8c2f1987-8965-449c-b426-b46515332392"
May 25 13:06:12.352869 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:12.352844    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vqnvc_d21bdf8b-d0ab-48f2-8736-54e92947dd53/ovn-acl-logging/0.log"
May 25 13:06:12.353381 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:12.353267    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vqnvc" event={"ID":"d21bdf8b-d0ab-48f2-8736-54e92947dd53","Type":"ContainerStarted","Data":"fd113451080948747cbb484020e4acb985f3aeefa2ed20d017e64766116e6192"}
May 25 13:06:12.779434 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:12.779397    2557 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-vwnkg"
May 25 13:06:12.780070 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:12.780053    2557 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-vwnkg"
May 25 13:06:13.145328 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:13.145296    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-96mr4"
May 25 13:06:13.145536 ip-10-0-143-235 kubenswrapper[2557]: E0525 13:06:13.145443    2557 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-96mr4" podUID="a8d6074e-8880-4fd2-b612-de33be9d873e"
May 25 13:06:14.146199 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:14.146014    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5lm4j"
May 25 13:06:14.146851 ip-10-0-143-235 kubenswrapper[2557]: E0525 13:06:14.146278    2557 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5lm4j" podUID="8c2f1987-8965-449c-b426-b46515332392"
May 25 13:06:14.359612 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:14.359581    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vqnvc_d21bdf8b-d0ab-48f2-8736-54e92947dd53/ovn-acl-logging/0.log"
May 25 13:06:14.359951 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:14.359930    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vqnvc" event={"ID":"d21bdf8b-d0ab-48f2-8736-54e92947dd53","Type":"ContainerStarted","Data":"a6b1ed6b0085d70b9135adb1775c703cbaec372f16d258af28b2f928e03dc93d"}
May 25 13:06:14.360332 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:14.360303    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-vqnvc"
May 25 13:06:14.360518 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:14.360501    2557 scope.go:117] "RemoveContainer" containerID="e97710c4274bdc36d4fa397c0a9bc04b0d71dc6365a4a445877d223957e22c0d"
May 25 13:06:14.361804 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:14.361776    2557 generic.go:358] "Generic (PLEG): container finished" podID="1dcac962-5867-4644-b3b3-ce62b3216b79" containerID="4b19ceb72d18313bf7958bfc01ee87ff95f534aca97cdb6c9d4f6b7336e3a360" exitCode=0
May 25 13:06:14.361881 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:14.361834    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lphh6" event={"ID":"1dcac962-5867-4644-b3b3-ce62b3216b79","Type":"ContainerDied","Data":"4b19ceb72d18313bf7958bfc01ee87ff95f534aca97cdb6c9d4f6b7336e3a360"}
May 25 13:06:14.378090 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:14.378061    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vqnvc"
May 25 13:06:15.145591 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:15.145552    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-96mr4"
May 25 13:06:15.145764 ip-10-0-143-235 kubenswrapper[2557]: E0525 13:06:15.145686    2557 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-96mr4" podUID="a8d6074e-8880-4fd2-b612-de33be9d873e"
May 25 13:06:15.365431 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:15.365398    2557 generic.go:358] "Generic (PLEG): container finished" podID="1dcac962-5867-4644-b3b3-ce62b3216b79" containerID="d15e751ecdabab65e6cdbe6b8b53e0b63d5158c1fc1e22ad557b77d37effa45e" exitCode=0
May 25 13:06:15.365863 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:15.365475    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lphh6" event={"ID":"1dcac962-5867-4644-b3b3-ce62b3216b79","Type":"ContainerDied","Data":"d15e751ecdabab65e6cdbe6b8b53e0b63d5158c1fc1e22ad557b77d37effa45e"}
May 25 13:06:15.369037 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:15.369019    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vqnvc_d21bdf8b-d0ab-48f2-8736-54e92947dd53/ovn-acl-logging/0.log"
May 25 13:06:15.369329 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:15.369300    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vqnvc" event={"ID":"d21bdf8b-d0ab-48f2-8736-54e92947dd53","Type":"ContainerStarted","Data":"5068cf9773a481b303f5849d661d0d94b3b35ac334a3d09ed83adb2122254b49"}
May 25 13:06:15.369599 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:15.369577    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-vqnvc"
May 25 13:06:15.369687 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:15.369603    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-vqnvc"
May 25 13:06:15.386685 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:15.386513    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vqnvc"
May 25 13:06:15.417477 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:15.417419    2557 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-vqnvc" podStartSLOduration=9.078332879 podStartE2EDuration="27.417398192s" podCreationTimestamp="2026-05-25 13:05:48 +0000 UTC" firstStartedPulling="2026-05-25 13:05:50.598237581 +0000 UTC m=+3.061253730" lastFinishedPulling="2026-05-25 13:06:08.937302908 +0000 UTC m=+21.400319043" observedRunningTime="2026-05-25 13:06:15.417269024 +0000 UTC m=+27.880285183" watchObservedRunningTime="2026-05-25 13:06:15.417398192 +0000 UTC m=+27.880414350"
May 25 13:06:15.693104 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:15.693021    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-5lm4j"]
May 25 13:06:15.693262 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:15.693154    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5lm4j"
May 25 13:06:15.693304 ip-10-0-143-235 kubenswrapper[2557]: E0525 13:06:15.693280    2557 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5lm4j" podUID="8c2f1987-8965-449c-b426-b46515332392"
May 25 13:06:15.696240 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:15.696218    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-96mr4"]
May 25 13:06:15.696366 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:15.696314    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-96mr4"
May 25 13:06:15.696439 ip-10-0-143-235 kubenswrapper[2557]: E0525 13:06:15.696421    2557 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-96mr4" podUID="a8d6074e-8880-4fd2-b612-de33be9d873e"
May 25 13:06:15.847318 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:15.847277    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-vwnkg"
May 25 13:06:15.847506 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:15.847489    2557 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
May 25 13:06:15.848165 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:15.848139    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-vwnkg"
May 25 13:06:16.373244 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:16.373208    2557 generic.go:358] "Generic (PLEG): container finished" podID="1dcac962-5867-4644-b3b3-ce62b3216b79" containerID="0fe48faad475ba3d0c5aa1fc54778afaffe1dfd0cecce8aafb3cc83cdfea4e19" exitCode=0
May 25 13:06:16.373796 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:16.373301    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lphh6" event={"ID":"1dcac962-5867-4644-b3b3-ce62b3216b79","Type":"ContainerDied","Data":"0fe48faad475ba3d0c5aa1fc54778afaffe1dfd0cecce8aafb3cc83cdfea4e19"}
May 25 13:06:17.145628 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:17.145592    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-96mr4"
May 25 13:06:17.145628 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:17.145626    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5lm4j"
May 25 13:06:17.145867 ip-10-0-143-235 kubenswrapper[2557]: E0525 13:06:17.145729    2557 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-96mr4" podUID="a8d6074e-8880-4fd2-b612-de33be9d873e"
May 25 13:06:17.145921 ip-10-0-143-235 kubenswrapper[2557]: E0525 13:06:17.145863    2557 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5lm4j" podUID="8c2f1987-8965-449c-b426-b46515332392"
May 25 13:06:19.145670 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:19.145622    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-96mr4"
May 25 13:06:19.146291 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:19.145623    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5lm4j"
May 25 13:06:19.146291 ip-10-0-143-235 kubenswrapper[2557]: E0525 13:06:19.145763    2557 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-96mr4" podUID="a8d6074e-8880-4fd2-b612-de33be9d873e"
May 25 13:06:19.146291 ip-10-0-143-235 kubenswrapper[2557]: E0525 13:06:19.145862    2557 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5lm4j" podUID="8c2f1987-8965-449c-b426-b46515332392"
May 25 13:06:19.883747 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:19.883720    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-hp2w9_50653f82-1790-4f22-94c1-406308112ca5/dns-node-resolver/0.log"
May 25 13:06:21.043785 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:21.043753    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-v9wvs_9d1a5d53-d761-478d-a85e-0974f65121e3/node-ca/0.log"
May 25 13:06:21.145705 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:21.145476    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5lm4j"
May 25 13:06:21.145919 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:21.145476    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-96mr4"
May 25 13:06:21.145919 ip-10-0-143-235 kubenswrapper[2557]: E0525 13:06:21.145844    2557 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5lm4j" podUID="8c2f1987-8965-449c-b426-b46515332392"
May 25 13:06:21.145919 ip-10-0-143-235 kubenswrapper[2557]: E0525 13:06:21.145905    2557 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-96mr4" podUID="a8d6074e-8880-4fd2-b612-de33be9d873e"
May 25 13:06:21.784979 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:21.784932    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c2f1987-8965-449c-b426-b46515332392-metrics-certs\") pod \"network-metrics-daemon-5lm4j\" (UID: \"8c2f1987-8965-449c-b426-b46515332392\") " pod="openshift-multus/network-metrics-daemon-5lm4j"
May 25 13:06:21.785162 ip-10-0-143-235 kubenswrapper[2557]: E0525 13:06:21.785132    2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 13:06:21.785218 ip-10-0-143-235 kubenswrapper[2557]: E0525 13:06:21.785207    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c2f1987-8965-449c-b426-b46515332392-metrics-certs podName:8c2f1987-8965-449c-b426-b46515332392 nodeName:}" failed. No retries permitted until 2026-05-25 13:06:53.785190932 +0000 UTC m=+66.248207066 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8c2f1987-8965-449c-b426-b46515332392-metrics-certs") pod "network-metrics-daemon-5lm4j" (UID: "8c2f1987-8965-449c-b426-b46515332392") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 13:06:21.885967 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:21.885927    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-4f7rk\" (UniqueName: \"kubernetes.io/projected/a8d6074e-8880-4fd2-b612-de33be9d873e-kube-api-access-4f7rk\") pod \"network-check-target-96mr4\" (UID: \"a8d6074e-8880-4fd2-b612-de33be9d873e\") " pod="openshift-network-diagnostics/network-check-target-96mr4"
May 25 13:06:21.886145 ip-10-0-143-235 kubenswrapper[2557]: E0525 13:06:21.886097    2557 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 25 13:06:21.886145 ip-10-0-143-235 kubenswrapper[2557]: E0525 13:06:21.886123    2557 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 25 13:06:21.886145 ip-10-0-143-235 kubenswrapper[2557]: E0525 13:06:21.886137    2557 projected.go:194] Error preparing data for projected volume kube-api-access-4f7rk for pod openshift-network-diagnostics/network-check-target-96mr4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 13:06:21.886280 ip-10-0-143-235 kubenswrapper[2557]: E0525 13:06:21.886198    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a8d6074e-8880-4fd2-b612-de33be9d873e-kube-api-access-4f7rk podName:a8d6074e-8880-4fd2-b612-de33be9d873e nodeName:}" failed. No retries permitted until 2026-05-25 13:06:53.886179092 +0000 UTC m=+66.349195240 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-4f7rk" (UniqueName: "kubernetes.io/projected/a8d6074e-8880-4fd2-b612-de33be9d873e-kube-api-access-4f7rk") pod "network-check-target-96mr4" (UID: "a8d6074e-8880-4fd2-b612-de33be9d873e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 13:06:23.145326 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:23.145289    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-96mr4"
May 25 13:06:23.145829 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:23.145289    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5lm4j"
May 25 13:06:23.145829 ip-10-0-143-235 kubenswrapper[2557]: E0525 13:06:23.145437    2557 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-96mr4" podUID="a8d6074e-8880-4fd2-b612-de33be9d873e"
May 25 13:06:23.145829 ip-10-0-143-235 kubenswrapper[2557]: E0525 13:06:23.145473    2557 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5lm4j" podUID="8c2f1987-8965-449c-b426-b46515332392"
May 25 13:06:23.391742 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:23.391707    2557 generic.go:358] "Generic (PLEG): container finished" podID="1dcac962-5867-4644-b3b3-ce62b3216b79" containerID="196f82a8c888b44420143eb6272cfb3dca6ebd7b2f3e4a48784534a57c981b85" exitCode=0
May 25 13:06:23.391906 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:23.391751    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lphh6" event={"ID":"1dcac962-5867-4644-b3b3-ce62b3216b79","Type":"ContainerDied","Data":"196f82a8c888b44420143eb6272cfb3dca6ebd7b2f3e4a48784534a57c981b85"}
May 25 13:06:24.395747 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:24.395716    2557 generic.go:358] "Generic (PLEG): container finished" podID="1dcac962-5867-4644-b3b3-ce62b3216b79" containerID="51572f017e68c14a8d5094659fc329bce10c9ff939d0c1180632f41bd8ee5da0" exitCode=0
May 25 13:06:24.396153 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:24.395757    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lphh6" event={"ID":"1dcac962-5867-4644-b3b3-ce62b3216b79","Type":"ContainerDied","Data":"51572f017e68c14a8d5094659fc329bce10c9ff939d0c1180632f41bd8ee5da0"}
May 25 13:06:25.146144 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:25.146106    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-96mr4"
May 25 13:06:25.146315 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:25.146106    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5lm4j"
May 25 13:06:25.146315 ip-10-0-143-235 kubenswrapper[2557]: E0525 13:06:25.146226    2557 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-96mr4" podUID="a8d6074e-8880-4fd2-b612-de33be9d873e"
May 25 13:06:25.146412 ip-10-0-143-235 kubenswrapper[2557]: E0525 13:06:25.146313    2557 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5lm4j" podUID="8c2f1987-8965-449c-b426-b46515332392"
May 25 13:06:25.400899 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:25.400812    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lphh6" event={"ID":"1dcac962-5867-4644-b3b3-ce62b3216b79","Type":"ContainerStarted","Data":"29daf6e3ba2cdb9f39f00416e041ecad6852a2c36068c842b2d39a62b2eb6b7c"}
May 25 13:06:25.426433 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:25.426379    2557 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-lphh6" podStartSLOduration=5.659255095 podStartE2EDuration="37.426362476s" podCreationTimestamp="2026-05-25 13:05:48 +0000 UTC" firstStartedPulling="2026-05-25 13:05:50.59099278 +0000 UTC m=+3.054008921" lastFinishedPulling="2026-05-25 13:06:22.358100151 +0000 UTC m=+34.821116302" observedRunningTime="2026-05-25 13:06:25.425976643 +0000 UTC m=+37.888992801" watchObservedRunningTime="2026-05-25 13:06:25.426362476 +0000 UTC m=+37.889378641"
May 25 13:06:27.146064 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:27.146024    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5lm4j"
May 25 13:06:27.146483 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:27.146022    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-96mr4"
May 25 13:06:27.146483 ip-10-0-143-235 kubenswrapper[2557]: E0525 13:06:27.146162    2557 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5lm4j" podUID="8c2f1987-8965-449c-b426-b46515332392"
May 25 13:06:27.146483 ip-10-0-143-235 kubenswrapper[2557]: E0525 13:06:27.146228    2557 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-96mr4" podUID="a8d6074e-8880-4fd2-b612-de33be9d873e"
May 25 13:06:29.146039 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:29.145994    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-96mr4"
May 25 13:06:29.146476 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:29.146002    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5lm4j"
May 25 13:06:29.146476 ip-10-0-143-235 kubenswrapper[2557]: E0525 13:06:29.146104    2557 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-96mr4" podUID="a8d6074e-8880-4fd2-b612-de33be9d873e"
May 25 13:06:29.146476 ip-10-0-143-235 kubenswrapper[2557]: E0525 13:06:29.146181    2557 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5lm4j" podUID="8c2f1987-8965-449c-b426-b46515332392"
May 25 13:06:31.145838 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:31.145802    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-96mr4"
May 25 13:06:31.146220 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:31.145808    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5lm4j"
May 25 13:06:31.146220 ip-10-0-143-235 kubenswrapper[2557]: E0525 13:06:31.145916    2557 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-96mr4" podUID="a8d6074e-8880-4fd2-b612-de33be9d873e"
May 25 13:06:31.146220 ip-10-0-143-235 kubenswrapper[2557]: E0525 13:06:31.145987    2557 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5lm4j" podUID="8c2f1987-8965-449c-b426-b46515332392"
May 25 13:06:33.145930 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:33.145887    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5lm4j"
May 25 13:06:33.146313 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:33.145888    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-96mr4"
May 25 13:06:33.146313 ip-10-0-143-235 kubenswrapper[2557]: E0525 13:06:33.146017    2557 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5lm4j" podUID="8c2f1987-8965-449c-b426-b46515332392"
May 25 13:06:33.146313 ip-10-0-143-235 kubenswrapper[2557]: E0525 13:06:33.146063    2557 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-96mr4" podUID="a8d6074e-8880-4fd2-b612-de33be9d873e"
May 25 13:06:35.146215 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:35.146174    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-96mr4"
May 25 13:06:35.146710 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:35.146228    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5lm4j"
May 25 13:06:35.146710 ip-10-0-143-235 kubenswrapper[2557]: E0525 13:06:35.146304    2557 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-96mr4" podUID="a8d6074e-8880-4fd2-b612-de33be9d873e"
May 25 13:06:35.146710 ip-10-0-143-235 kubenswrapper[2557]: E0525 13:06:35.146427    2557 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5lm4j" podUID="8c2f1987-8965-449c-b426-b46515332392"
May 25 13:06:37.145459 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:37.145426    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5lm4j"
May 25 13:06:37.145844 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:37.145426    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-96mr4"
May 25 13:06:37.145844 ip-10-0-143-235 kubenswrapper[2557]: E0525 13:06:37.145539    2557 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5lm4j" podUID="8c2f1987-8965-449c-b426-b46515332392"
May 25 13:06:37.145844 ip-10-0-143-235 kubenswrapper[2557]: E0525 13:06:37.145604    2557 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-96mr4" podUID="a8d6074e-8880-4fd2-b612-de33be9d873e"
May 25 13:06:39.146188 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:39.146142    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-96mr4"
May 25 13:06:39.146723 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:39.146155    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5lm4j"
May 25 13:06:39.146723 ip-10-0-143-235 kubenswrapper[2557]: E0525 13:06:39.146245    2557 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-96mr4" podUID="a8d6074e-8880-4fd2-b612-de33be9d873e"
May 25 13:06:39.146723 ip-10-0-143-235 kubenswrapper[2557]: E0525 13:06:39.146398    2557 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5lm4j" podUID="8c2f1987-8965-449c-b426-b46515332392"
May 25 13:06:41.145214 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:41.145175    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5lm4j"
May 25 13:06:41.145720 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:41.145175    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-96mr4"
May 25 13:06:41.145720 ip-10-0-143-235 kubenswrapper[2557]: E0525 13:06:41.145319    2557 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5lm4j" podUID="8c2f1987-8965-449c-b426-b46515332392"
May 25 13:06:41.145720 ip-10-0-143-235 kubenswrapper[2557]: E0525 13:06:41.145428    2557 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-96mr4" podUID="a8d6074e-8880-4fd2-b612-de33be9d873e"
May 25 13:06:42.314474 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:42.314446    2557 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-143-235.ec2.internal" event="NodeReady"
May 25 13:06:42.314838 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:42.314560    2557 kubelet_node_status.go:550] "Fast updating node status as it just became ready"
May 25 13:06:42.363667 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:42.363633    2557 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-5cwgw"]
May 25 13:06:42.365425 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:42.365408    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-5cwgw"
May 25 13:06:42.367386 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:42.367365    2557 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-glgst"]
May 25 13:06:42.368434 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:42.368416    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
May 25 13:06:42.368530 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:42.368454    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
May 25 13:06:42.368603 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:42.368545    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-5d4pm\""
May 25 13:06:42.369497 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:42.369480    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-glgst"
May 25 13:06:42.372503 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:42.372483    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
May 25 13:06:42.372576 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:42.372526    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
May 25 13:06:42.373328 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:42.373229    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
May 25 13:06:42.373820 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:42.373805    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-wgqp9\""
May 25 13:06:42.373913 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:42.373817    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
May 25 13:06:42.382964 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:42.382940    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-5cwgw"]
May 25 13:06:42.389076 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:42.389052    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-glgst"]
May 25 13:06:42.431819 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:42.431713    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4f20140e-81a7-4c76-9daa-eaa7057ba3d3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-glgst\" (UID: \"4f20140e-81a7-4c76-9daa-eaa7057ba3d3\") " pod="openshift-insights/insights-runtime-extractor-glgst"
May 25 13:06:42.431819 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:42.431764    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4f20140e-81a7-4c76-9daa-eaa7057ba3d3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-glgst\" (UID: \"4f20140e-81a7-4c76-9daa-eaa7057ba3d3\") " pod="openshift-insights/insights-runtime-extractor-glgst"
May 25 13:06:42.432070 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:42.431870    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4f20140e-81a7-4c76-9daa-eaa7057ba3d3-crio-socket\") pod \"insights-runtime-extractor-glgst\" (UID: \"4f20140e-81a7-4c76-9daa-eaa7057ba3d3\") " pod="openshift-insights/insights-runtime-extractor-glgst"
May 25 13:06:42.432070 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:42.431905    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/4f20140e-81a7-4c76-9daa-eaa7057ba3d3-data-volume\") pod \"insights-runtime-extractor-glgst\" (UID: \"4f20140e-81a7-4c76-9daa-eaa7057ba3d3\") " pod="openshift-insights/insights-runtime-extractor-glgst"
May 25 13:06:42.432070 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:42.431942    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/776386fc-900b-494b-a190-cb092725bbf7-metrics-tls\") pod \"dns-default-5cwgw\" (UID: \"776386fc-900b-494b-a190-cb092725bbf7\") " pod="openshift-dns/dns-default-5cwgw"
May 25 13:06:42.432070 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:42.431982    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/776386fc-900b-494b-a190-cb092725bbf7-config-volume\") pod \"dns-default-5cwgw\" (UID: \"776386fc-900b-494b-a190-cb092725bbf7\") " pod="openshift-dns/dns-default-5cwgw"
May 25 13:06:42.432070 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:42.432045    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfh8t\" (UniqueName: \"kubernetes.io/projected/4f20140e-81a7-4c76-9daa-eaa7057ba3d3-kube-api-access-sfh8t\") pod \"insights-runtime-extractor-glgst\" (UID: \"4f20140e-81a7-4c76-9daa-eaa7057ba3d3\") " pod="openshift-insights/insights-runtime-extractor-glgst"
May 25 13:06:42.432070 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:42.432067    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/776386fc-900b-494b-a190-cb092725bbf7-tmp-dir\") pod \"dns-default-5cwgw\" (UID: \"776386fc-900b-494b-a190-cb092725bbf7\") " pod="openshift-dns/dns-default-5cwgw"
May 25 13:06:42.432282 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:42.432093    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rcjds\" (UniqueName: \"kubernetes.io/projected/776386fc-900b-494b-a190-cb092725bbf7-kube-api-access-rcjds\") pod \"dns-default-5cwgw\" (UID: \"776386fc-900b-494b-a190-cb092725bbf7\") " pod="openshift-dns/dns-default-5cwgw"
May 25 13:06:42.468531 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:42.468499    2557 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-5thz9"]
May 25 13:06:42.470462 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:42.470447    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-5thz9"
May 25 13:06:42.473271 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:42.473239    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
May 25 13:06:42.473271 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:42.473253    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-vbzlk\""
May 25 13:06:42.473472 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:42.473322    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
May 25 13:06:42.473472 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:42.473323    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
May 25 13:06:42.480984 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:42.480959    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-5thz9"]
May 25 13:06:42.532522 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:42.532485    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4f20140e-81a7-4c76-9daa-eaa7057ba3d3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-glgst\" (UID: \"4f20140e-81a7-4c76-9daa-eaa7057ba3d3\") " pod="openshift-insights/insights-runtime-extractor-glgst"
May 25 13:06:42.532522 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:42.532527    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b8ace320-b067-48f1-9a9d-9f0681b3ee25-cert\") pod \"ingress-canary-5thz9\" (UID: \"b8ace320-b067-48f1-9a9d-9f0681b3ee25\") " pod="openshift-ingress-canary/ingress-canary-5thz9"
May 25 13:06:42.532744 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:42.532558    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4f20140e-81a7-4c76-9daa-eaa7057ba3d3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-glgst\" (UID: \"4f20140e-81a7-4c76-9daa-eaa7057ba3d3\") " pod="openshift-insights/insights-runtime-extractor-glgst"
May 25 13:06:42.532744 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:42.532618    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4f20140e-81a7-4c76-9daa-eaa7057ba3d3-crio-socket\") pod \"insights-runtime-extractor-glgst\" (UID: \"4f20140e-81a7-4c76-9daa-eaa7057ba3d3\") " pod="openshift-insights/insights-runtime-extractor-glgst"
May 25 13:06:42.532744 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:42.532648    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/4f20140e-81a7-4c76-9daa-eaa7057ba3d3-data-volume\") pod \"insights-runtime-extractor-glgst\" (UID: \"4f20140e-81a7-4c76-9daa-eaa7057ba3d3\") " pod="openshift-insights/insights-runtime-extractor-glgst"
May 25 13:06:42.532744 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:42.532674    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/776386fc-900b-494b-a190-cb092725bbf7-metrics-tls\") pod \"dns-default-5cwgw\" (UID: \"776386fc-900b-494b-a190-cb092725bbf7\") " pod="openshift-dns/dns-default-5cwgw"
May 25 13:06:42.532744 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:42.532700    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4f20140e-81a7-4c76-9daa-eaa7057ba3d3-crio-socket\") pod \"insights-runtime-extractor-glgst\" (UID: \"4f20140e-81a7-4c76-9daa-eaa7057ba3d3\") " pod="openshift-insights/insights-runtime-extractor-glgst"
May 25 13:06:42.532744 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:42.532708    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/776386fc-900b-494b-a190-cb092725bbf7-config-volume\") pod \"dns-default-5cwgw\" (UID: \"776386fc-900b-494b-a190-cb092725bbf7\") " pod="openshift-dns/dns-default-5cwgw"
May 25 13:06:42.533022 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:42.532756    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-767kq\" (UniqueName: \"kubernetes.io/projected/b8ace320-b067-48f1-9a9d-9f0681b3ee25-kube-api-access-767kq\") pod \"ingress-canary-5thz9\" (UID: \"b8ace320-b067-48f1-9a9d-9f0681b3ee25\") " pod="openshift-ingress-canary/ingress-canary-5thz9"
May 25 13:06:42.533022 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:42.532810    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-sfh8t\" (UniqueName: \"kubernetes.io/projected/4f20140e-81a7-4c76-9daa-eaa7057ba3d3-kube-api-access-sfh8t\") pod \"insights-runtime-extractor-glgst\" (UID: \"4f20140e-81a7-4c76-9daa-eaa7057ba3d3\") " pod="openshift-insights/insights-runtime-extractor-glgst"
May 25 13:06:42.533022 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:42.532835    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/776386fc-900b-494b-a190-cb092725bbf7-tmp-dir\") pod \"dns-default-5cwgw\" (UID: \"776386fc-900b-494b-a190-cb092725bbf7\") " pod="openshift-dns/dns-default-5cwgw"
May 25 13:06:42.533022 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:42.532862    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-rcjds\" (UniqueName: \"kubernetes.io/projected/776386fc-900b-494b-a190-cb092725bbf7-kube-api-access-rcjds\") pod \"dns-default-5cwgw\" (UID: \"776386fc-900b-494b-a190-cb092725bbf7\") " pod="openshift-dns/dns-default-5cwgw"
May 25 13:06:42.533203 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:42.533034    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/4f20140e-81a7-4c76-9daa-eaa7057ba3d3-data-volume\") pod \"insights-runtime-extractor-glgst\" (UID: \"4f20140e-81a7-4c76-9daa-eaa7057ba3d3\") " pod="openshift-insights/insights-runtime-extractor-glgst"
May 25 13:06:42.533271 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:42.533249    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4f20140e-81a7-4c76-9daa-eaa7057ba3d3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-glgst\" (UID: \"4f20140e-81a7-4c76-9daa-eaa7057ba3d3\") " pod="openshift-insights/insights-runtime-extractor-glgst"
May 25 13:06:42.533326 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:42.533249    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/776386fc-900b-494b-a190-cb092725bbf7-config-volume\") pod \"dns-default-5cwgw\" (UID: \"776386fc-900b-494b-a190-cb092725bbf7\") " pod="openshift-dns/dns-default-5cwgw"
May 25 13:06:42.533326 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:42.533260    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/776386fc-900b-494b-a190-cb092725bbf7-tmp-dir\") pod \"dns-default-5cwgw\" (UID: \"776386fc-900b-494b-a190-cb092725bbf7\") " pod="openshift-dns/dns-default-5cwgw"
May 25 13:06:42.536767 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:42.536733    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/776386fc-900b-494b-a190-cb092725bbf7-metrics-tls\") pod \"dns-default-5cwgw\" (UID: \"776386fc-900b-494b-a190-cb092725bbf7\") " pod="openshift-dns/dns-default-5cwgw"
May 25 13:06:42.536902 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:42.536733    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4f20140e-81a7-4c76-9daa-eaa7057ba3d3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-glgst\" (UID: \"4f20140e-81a7-4c76-9daa-eaa7057ba3d3\") " pod="openshift-insights/insights-runtime-extractor-glgst"
May 25 13:06:42.541525 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:42.541488    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rcjds\" (UniqueName: \"kubernetes.io/projected/776386fc-900b-494b-a190-cb092725bbf7-kube-api-access-rcjds\") pod \"dns-default-5cwgw\" (UID: \"776386fc-900b-494b-a190-cb092725bbf7\") " pod="openshift-dns/dns-default-5cwgw"
May 25 13:06:42.541668 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:42.541567    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfh8t\" (UniqueName: \"kubernetes.io/projected/4f20140e-81a7-4c76-9daa-eaa7057ba3d3-kube-api-access-sfh8t\") pod \"insights-runtime-extractor-glgst\" (UID: \"4f20140e-81a7-4c76-9daa-eaa7057ba3d3\") " pod="openshift-insights/insights-runtime-extractor-glgst"
May 25 13:06:42.633256 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:42.633215    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-767kq\" (UniqueName: \"kubernetes.io/projected/b8ace320-b067-48f1-9a9d-9f0681b3ee25-kube-api-access-767kq\") pod \"ingress-canary-5thz9\" (UID: \"b8ace320-b067-48f1-9a9d-9f0681b3ee25\") " pod="openshift-ingress-canary/ingress-canary-5thz9"
May 25 13:06:42.633465 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:42.633273    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b8ace320-b067-48f1-9a9d-9f0681b3ee25-cert\") pod \"ingress-canary-5thz9\" (UID: \"b8ace320-b067-48f1-9a9d-9f0681b3ee25\") " pod="openshift-ingress-canary/ingress-canary-5thz9"
May 25 13:06:42.635761 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:42.635739    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b8ace320-b067-48f1-9a9d-9f0681b3ee25-cert\") pod \"ingress-canary-5thz9\" (UID: \"b8ace320-b067-48f1-9a9d-9f0681b3ee25\") " pod="openshift-ingress-canary/ingress-canary-5thz9"
May 25 13:06:42.641706 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:42.641681    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-767kq\" (UniqueName: \"kubernetes.io/projected/b8ace320-b067-48f1-9a9d-9f0681b3ee25-kube-api-access-767kq\") pod \"ingress-canary-5thz9\" (UID: \"b8ace320-b067-48f1-9a9d-9f0681b3ee25\") " pod="openshift-ingress-canary/ingress-canary-5thz9"
May 25 13:06:42.676513 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:42.676480    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-5cwgw"
May 25 13:06:42.681254 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:42.681229    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-glgst"
May 25 13:06:42.778778 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:42.778751    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-5thz9"
May 25 13:06:42.827498 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:42.827467    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-5cwgw"]
May 25 13:06:42.836611 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:06:42.836251    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod776386fc_900b_494b_a190_cb092725bbf7.slice/crio-5a475665f0a9b4e1c46b669425463730a01530f50c308421faf47b6cd29c1093 WatchSource:0}: Error finding container 5a475665f0a9b4e1c46b669425463730a01530f50c308421faf47b6cd29c1093: Status 404 returned error can't find the container with id 5a475665f0a9b4e1c46b669425463730a01530f50c308421faf47b6cd29c1093
May 25 13:06:42.849372 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:42.845769    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-glgst"]
May 25 13:06:42.864673 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:06:42.864555    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f20140e_81a7_4c76_9daa_eaa7057ba3d3.slice/crio-2be0f425d2991a310627bcc63a567d90413a3d905504de8adf3dc0d8f8dab9bd WatchSource:0}: Error finding container 2be0f425d2991a310627bcc63a567d90413a3d905504de8adf3dc0d8f8dab9bd: Status 404 returned error can't find the container with id 2be0f425d2991a310627bcc63a567d90413a3d905504de8adf3dc0d8f8dab9bd
May 25 13:06:42.931708 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:42.931613    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-5thz9"]
May 25 13:06:42.934077 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:06:42.934048    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8ace320_b067_48f1_9a9d_9f0681b3ee25.slice/crio-738a34b1e45d40bc7a91b50a870d6820a157f8f2c5349f624e5a724fc5be0bf6 WatchSource:0}: Error finding container 738a34b1e45d40bc7a91b50a870d6820a157f8f2c5349f624e5a724fc5be0bf6: Status 404 returned error can't find the container with id 738a34b1e45d40bc7a91b50a870d6820a157f8f2c5349f624e5a724fc5be0bf6
May 25 13:06:43.145653 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:43.145619    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5lm4j"
May 25 13:06:43.145853 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:43.145622    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-96mr4"
May 25 13:06:43.149154 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:43.149130    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
May 25 13:06:43.149303 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:43.149282    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-gcddr\""
May 25 13:06:43.149415 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:43.149398    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-clhmx\""
May 25 13:06:43.149473 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:43.149449    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
May 25 13:06:43.150315 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:43.150301    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
May 25 13:06:43.437396 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:43.437360    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-glgst" event={"ID":"4f20140e-81a7-4c76-9daa-eaa7057ba3d3","Type":"ContainerStarted","Data":"5e70dcf18eb295452a6f5ca8fe64d7657719c240a67aeb25a121f9b517d3d72e"}
May 25 13:06:43.437891 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:43.437408    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-glgst" event={"ID":"4f20140e-81a7-4c76-9daa-eaa7057ba3d3","Type":"ContainerStarted","Data":"2be0f425d2991a310627bcc63a567d90413a3d905504de8adf3dc0d8f8dab9bd"}
May 25 13:06:43.439164 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:43.439101    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-5thz9" event={"ID":"b8ace320-b067-48f1-9a9d-9f0681b3ee25","Type":"ContainerStarted","Data":"738a34b1e45d40bc7a91b50a870d6820a157f8f2c5349f624e5a724fc5be0bf6"}
May 25 13:06:43.441381 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:43.441329    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-5cwgw" event={"ID":"776386fc-900b-494b-a190-cb092725bbf7","Type":"ContainerStarted","Data":"5a475665f0a9b4e1c46b669425463730a01530f50c308421faf47b6cd29c1093"}
May 25 13:06:44.445743 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:44.445698    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-glgst" event={"ID":"4f20140e-81a7-4c76-9daa-eaa7057ba3d3","Type":"ContainerStarted","Data":"52d1dfbbad7376de624ec3410330a06ac4d55f6205984bfadf482f39ba3b2581"}
May 25 13:06:45.383110 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:45.383075    2557 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-fnll9"]
May 25 13:06:45.416593 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:45.416506    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-fnll9"]
May 25 13:06:45.416734 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:45.416601    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-fnll9"
May 25 13:06:45.420029 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:45.419977    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
May 25 13:06:45.420168 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:45.420051    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-6fml5\""
May 25 13:06:45.463272 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:45.463224    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/91f38fd6-9e63-4273-8664-9f66efddcc5a-tls-certificates\") pod \"prometheus-operator-admission-webhook-6c4bbc67b7-fnll9\" (UID: \"91f38fd6-9e63-4273-8664-9f66efddcc5a\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-fnll9"
May 25 13:06:45.563716 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:45.563677    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/91f38fd6-9e63-4273-8664-9f66efddcc5a-tls-certificates\") pod \"prometheus-operator-admission-webhook-6c4bbc67b7-fnll9\" (UID: \"91f38fd6-9e63-4273-8664-9f66efddcc5a\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-fnll9"
May 25 13:06:45.563883 ip-10-0-143-235 kubenswrapper[2557]: E0525 13:06:45.563823    2557 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found
May 25 13:06:45.563926 ip-10-0-143-235 kubenswrapper[2557]: E0525 13:06:45.563891    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/91f38fd6-9e63-4273-8664-9f66efddcc5a-tls-certificates podName:91f38fd6-9e63-4273-8664-9f66efddcc5a nodeName:}" failed. No retries permitted until 2026-05-25 13:06:46.063874853 +0000 UTC m=+58.526890989 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/91f38fd6-9e63-4273-8664-9f66efddcc5a-tls-certificates") pod "prometheus-operator-admission-webhook-6c4bbc67b7-fnll9" (UID: "91f38fd6-9e63-4273-8664-9f66efddcc5a") : secret "prometheus-operator-admission-webhook-tls" not found
May 25 13:06:46.068310 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:46.068265    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/91f38fd6-9e63-4273-8664-9f66efddcc5a-tls-certificates\") pod \"prometheus-operator-admission-webhook-6c4bbc67b7-fnll9\" (UID: \"91f38fd6-9e63-4273-8664-9f66efddcc5a\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-fnll9"
May 25 13:06:46.070914 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:46.070885    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/91f38fd6-9e63-4273-8664-9f66efddcc5a-tls-certificates\") pod \"prometheus-operator-admission-webhook-6c4bbc67b7-fnll9\" (UID: \"91f38fd6-9e63-4273-8664-9f66efddcc5a\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-fnll9"
May 25 13:06:46.327123 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:46.327094    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-fnll9"
May 25 13:06:46.452629 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:46.452595    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-5thz9" event={"ID":"b8ace320-b067-48f1-9a9d-9f0681b3ee25","Type":"ContainerStarted","Data":"1f91678517ca4ea4046154dda7824af7bd7164095902da1402012200f6648009"}
May 25 13:06:46.454235 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:46.454186    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-5cwgw" event={"ID":"776386fc-900b-494b-a190-cb092725bbf7","Type":"ContainerStarted","Data":"d70071d555da6a9b061c5db00bf7dfbbf6ad5bb96e22a3845f2d4d1bf722e2bb"}
May 25 13:06:46.454235 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:46.454222    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-5cwgw" event={"ID":"776386fc-900b-494b-a190-cb092725bbf7","Type":"ContainerStarted","Data":"ae74c4adce9de90f7d6e62054d9db8b025b3b3940d7b38f1281656d283e6fda9"}
May 25 13:06:46.454466 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:46.454319    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-5cwgw"
May 25 13:06:46.455784 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:46.455759    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-fnll9"]
May 25 13:06:46.456577 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:46.456464    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-glgst" event={"ID":"4f20140e-81a7-4c76-9daa-eaa7057ba3d3","Type":"ContainerStarted","Data":"d8412cfa9e72e248c9c199aa05312c7ece893e38b7dcbeffe5147a34a3e36c95"}
May 25 13:06:46.459069 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:06:46.459046    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91f38fd6_9e63_4273_8664_9f66efddcc5a.slice/crio-af27660aa49a8e0c05e59c9bc2205fb9aff019f19889b71167aec476aeb7a913 WatchSource:0}: Error finding container af27660aa49a8e0c05e59c9bc2205fb9aff019f19889b71167aec476aeb7a913: Status 404 returned error can't find the container with id af27660aa49a8e0c05e59c9bc2205fb9aff019f19889b71167aec476aeb7a913
May 25 13:06:46.470747 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:46.470703    2557 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-5thz9" podStartSLOduration=1.783268667 podStartE2EDuration="4.470688217s" podCreationTimestamp="2026-05-25 13:06:42 +0000 UTC" firstStartedPulling="2026-05-25 13:06:42.935918877 +0000 UTC m=+55.398935013" lastFinishedPulling="2026-05-25 13:06:45.623338427 +0000 UTC m=+58.086354563" observedRunningTime="2026-05-25 13:06:46.470676421 +0000 UTC m=+58.933692591" watchObservedRunningTime="2026-05-25 13:06:46.470688217 +0000 UTC m=+58.933704372"
May 25 13:06:46.488728 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:46.488680    2557 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-dns/dns-default-5cwgw" podStartSLOduration=1.708851624 podStartE2EDuration="4.488665128s" podCreationTimestamp="2026-05-25 13:06:42 +0000 UTC" firstStartedPulling="2026-05-25 13:06:42.837664103 +0000 UTC m=+55.300680237" lastFinishedPulling="2026-05-25 13:06:45.617477605 +0000 UTC m=+58.080493741" observedRunningTime="2026-05-25 13:06:46.488114889 +0000 UTC m=+58.951131058" watchObservedRunningTime="2026-05-25 13:06:46.488665128 +0000 UTC m=+58.951681285"
May 25 13:06:47.205632 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:47.205365    2557 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-glgst" podStartSLOduration=2.26480891 podStartE2EDuration="5.205327008s" podCreationTimestamp="2026-05-25 13:06:42 +0000 UTC" firstStartedPulling="2026-05-25 13:06:42.94650711 +0000 UTC m=+55.409523248" lastFinishedPulling="2026-05-25 13:06:45.887025195 +0000 UTC m=+58.350041346" observedRunningTime="2026-05-25 13:06:46.509061389 +0000 UTC m=+58.972077542" watchObservedRunningTime="2026-05-25 13:06:47.205327008 +0000 UTC m=+59.668343168"
May 25 13:06:47.206240 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:47.206218    2557 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-console/console-5594bdbb97-l255d"]
May 25 13:06:47.208324 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:47.208305    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5594bdbb97-l255d"
May 25 13:06:47.211163 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:47.211134    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
May 25 13:06:47.211279 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:47.211177    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
May 25 13:06:47.211362 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:47.211318    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
May 25 13:06:47.211362 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:47.211335    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
May 25 13:06:47.212627 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:47.212571    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
May 25 13:06:47.212744 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:47.212633    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
May 25 13:06:47.212807 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:47.212770    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-qkg9t\""
May 25 13:06:47.212892 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:47.212873    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
May 25 13:06:47.218360 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:47.218323    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
May 25 13:06:47.219376 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:47.219308    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5594bdbb97-l255d"]
May 25 13:06:47.278852 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:47.278814    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4bb7cd7c-97a8-467d-b7d5-e1db2943618f-console-oauth-config\") pod \"console-5594bdbb97-l255d\" (UID: \"4bb7cd7c-97a8-467d-b7d5-e1db2943618f\") " pod="openshift-console/console-5594bdbb97-l255d"
May 25 13:06:47.278852 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:47.278863    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4bb7cd7c-97a8-467d-b7d5-e1db2943618f-oauth-serving-cert\") pod \"console-5594bdbb97-l255d\" (UID: \"4bb7cd7c-97a8-467d-b7d5-e1db2943618f\") " pod="openshift-console/console-5594bdbb97-l255d"
May 25 13:06:47.279112 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:47.278993    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4bb7cd7c-97a8-467d-b7d5-e1db2943618f-trusted-ca-bundle\") pod \"console-5594bdbb97-l255d\" (UID: \"4bb7cd7c-97a8-467d-b7d5-e1db2943618f\") " pod="openshift-console/console-5594bdbb97-l255d"
May 25 13:06:47.279112 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:47.279026    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4bb7cd7c-97a8-467d-b7d5-e1db2943618f-console-serving-cert\") pod \"console-5594bdbb97-l255d\" (UID: \"4bb7cd7c-97a8-467d-b7d5-e1db2943618f\") " pod="openshift-console/console-5594bdbb97-l255d"
May 25 13:06:47.279112 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:47.279053    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4bb7cd7c-97a8-467d-b7d5-e1db2943618f-console-config\") pod \"console-5594bdbb97-l255d\" (UID: \"4bb7cd7c-97a8-467d-b7d5-e1db2943618f\") " pod="openshift-console/console-5594bdbb97-l255d"
May 25 13:06:47.279112 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:47.279077    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4bb7cd7c-97a8-467d-b7d5-e1db2943618f-service-ca\") pod \"console-5594bdbb97-l255d\" (UID: \"4bb7cd7c-97a8-467d-b7d5-e1db2943618f\") " pod="openshift-console/console-5594bdbb97-l255d"
May 25 13:06:47.279112 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:47.279102    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vxplw\" (UniqueName: \"kubernetes.io/projected/4bb7cd7c-97a8-467d-b7d5-e1db2943618f-kube-api-access-vxplw\") pod \"console-5594bdbb97-l255d\" (UID: \"4bb7cd7c-97a8-467d-b7d5-e1db2943618f\") " pod="openshift-console/console-5594bdbb97-l255d"
May 25 13:06:47.379710 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:47.379658    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4bb7cd7c-97a8-467d-b7d5-e1db2943618f-console-oauth-config\") pod \"console-5594bdbb97-l255d\" (UID: \"4bb7cd7c-97a8-467d-b7d5-e1db2943618f\") " pod="openshift-console/console-5594bdbb97-l255d"
May 25 13:06:47.379710 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:47.379712    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4bb7cd7c-97a8-467d-b7d5-e1db2943618f-oauth-serving-cert\") pod \"console-5594bdbb97-l255d\" (UID: \"4bb7cd7c-97a8-467d-b7d5-e1db2943618f\") " pod="openshift-console/console-5594bdbb97-l255d"
May 25 13:06:47.379892 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:47.379787    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4bb7cd7c-97a8-467d-b7d5-e1db2943618f-trusted-ca-bundle\") pod \"console-5594bdbb97-l255d\" (UID: \"4bb7cd7c-97a8-467d-b7d5-e1db2943618f\") " pod="openshift-console/console-5594bdbb97-l255d"
May 25 13:06:47.379892 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:47.379824    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4bb7cd7c-97a8-467d-b7d5-e1db2943618f-console-serving-cert\") pod \"console-5594bdbb97-l255d\" (UID: \"4bb7cd7c-97a8-467d-b7d5-e1db2943618f\") " pod="openshift-console/console-5594bdbb97-l255d"
May 25 13:06:47.379892 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:47.379852    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4bb7cd7c-97a8-467d-b7d5-e1db2943618f-console-config\") pod \"console-5594bdbb97-l255d\" (UID: \"4bb7cd7c-97a8-467d-b7d5-e1db2943618f\") " pod="openshift-console/console-5594bdbb97-l255d"
May 25 13:06:47.379892 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:47.379877    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4bb7cd7c-97a8-467d-b7d5-e1db2943618f-service-ca\") pod \"console-5594bdbb97-l255d\" (UID: \"4bb7cd7c-97a8-467d-b7d5-e1db2943618f\") " pod="openshift-console/console-5594bdbb97-l255d"
May 25 13:06:47.380062 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:47.379893    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-vxplw\" (UniqueName: \"kubernetes.io/projected/4bb7cd7c-97a8-467d-b7d5-e1db2943618f-kube-api-access-vxplw\") pod \"console-5594bdbb97-l255d\" (UID: \"4bb7cd7c-97a8-467d-b7d5-e1db2943618f\") " pod="openshift-console/console-5594bdbb97-l255d"
May 25 13:06:47.380884 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:47.380853    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4bb7cd7c-97a8-467d-b7d5-e1db2943618f-console-config\") pod \"console-5594bdbb97-l255d\" (UID: \"4bb7cd7c-97a8-467d-b7d5-e1db2943618f\") " pod="openshift-console/console-5594bdbb97-l255d"
May 25 13:06:47.381441 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:47.381420    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4bb7cd7c-97a8-467d-b7d5-e1db2943618f-trusted-ca-bundle\") pod \"console-5594bdbb97-l255d\" (UID: \"4bb7cd7c-97a8-467d-b7d5-e1db2943618f\") " pod="openshift-console/console-5594bdbb97-l255d"
May 25 13:06:47.381616 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:47.381427    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4bb7cd7c-97a8-467d-b7d5-e1db2943618f-oauth-serving-cert\") pod \"console-5594bdbb97-l255d\" (UID: \"4bb7cd7c-97a8-467d-b7d5-e1db2943618f\") " pod="openshift-console/console-5594bdbb97-l255d"
May 25 13:06:47.381616 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:47.381609    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4bb7cd7c-97a8-467d-b7d5-e1db2943618f-service-ca\") pod \"console-5594bdbb97-l255d\" (UID: \"4bb7cd7c-97a8-467d-b7d5-e1db2943618f\") " pod="openshift-console/console-5594bdbb97-l255d"
May 25 13:06:47.382723 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:47.382698    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4bb7cd7c-97a8-467d-b7d5-e1db2943618f-console-oauth-config\") pod \"console-5594bdbb97-l255d\" (UID: \"4bb7cd7c-97a8-467d-b7d5-e1db2943618f\") " pod="openshift-console/console-5594bdbb97-l255d"
May 25 13:06:47.382839 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:47.382706    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4bb7cd7c-97a8-467d-b7d5-e1db2943618f-console-serving-cert\") pod \"console-5594bdbb97-l255d\" (UID: \"4bb7cd7c-97a8-467d-b7d5-e1db2943618f\") " pod="openshift-console/console-5594bdbb97-l255d"
May 25 13:06:47.387879 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:47.387851    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vqnvc"
May 25 13:06:47.422200 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:47.422165    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxplw\" (UniqueName: \"kubernetes.io/projected/4bb7cd7c-97a8-467d-b7d5-e1db2943618f-kube-api-access-vxplw\") pod \"console-5594bdbb97-l255d\" (UID: \"4bb7cd7c-97a8-467d-b7d5-e1db2943618f\") " pod="openshift-console/console-5594bdbb97-l255d"
May 25 13:06:47.460136 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:47.460039    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-fnll9" event={"ID":"91f38fd6-9e63-4273-8664-9f66efddcc5a","Type":"ContainerStarted","Data":"7d97624163665bb4fa035ef891e8e521e4d26c70cc2a6111a3bc713017bd5eb5"}
May 25 13:06:47.460136 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:47.460085    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-fnll9" event={"ID":"91f38fd6-9e63-4273-8664-9f66efddcc5a","Type":"ContainerStarted","Data":"af27660aa49a8e0c05e59c9bc2205fb9aff019f19889b71167aec476aeb7a913"}
May 25 13:06:47.478621 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:47.478569    2557 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-fnll9" podStartSLOduration=1.6242726410000001 podStartE2EDuration="2.47855595s" podCreationTimestamp="2026-05-25 13:06:45 +0000 UTC" firstStartedPulling="2026-05-25 13:06:46.46073178 +0000 UTC m=+58.923747915" lastFinishedPulling="2026-05-25 13:06:47.315015074 +0000 UTC m=+59.778031224" observedRunningTime="2026-05-25 13:06:47.477272239 +0000 UTC m=+59.940288398" watchObservedRunningTime="2026-05-25 13:06:47.47855595 +0000 UTC m=+59.941572139"
May 25 13:06:47.520283 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:47.520241    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5594bdbb97-l255d"
May 25 13:06:47.645290 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:47.645256    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5594bdbb97-l255d"]
May 25 13:06:47.648221 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:06:47.648193    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4bb7cd7c_97a8_467d_b7d5_e1db2943618f.slice/crio-f39e482e90bff4b099e7e59f1e5780dac0e015c350b6a5a7d15b6a273e78c624 WatchSource:0}: Error finding container f39e482e90bff4b099e7e59f1e5780dac0e015c350b6a5a7d15b6a273e78c624: Status 404 returned error can't find the container with id f39e482e90bff4b099e7e59f1e5780dac0e015c350b6a5a7d15b6a273e78c624
May 25 13:06:48.463628 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:48.463588    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5594bdbb97-l255d" event={"ID":"4bb7cd7c-97a8-467d-b7d5-e1db2943618f","Type":"ContainerStarted","Data":"f39e482e90bff4b099e7e59f1e5780dac0e015c350b6a5a7d15b6a273e78c624"}
May 25 13:06:48.463935 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:48.463912    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-fnll9"
May 25 13:06:48.470416 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:48.470390    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-fnll9"
May 25 13:06:49.445442 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:49.445408    2557 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-7779bbdd78-vh2bs"]
May 25 13:06:49.447650 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:49.447627    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-7779bbdd78-vh2bs"
May 25 13:06:49.451977 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:49.451795    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
May 25 13:06:49.451977 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:49.451795    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
May 25 13:06:49.451977 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:49.451862    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-lnq4t\""
May 25 13:06:49.451977 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:49.451869    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
May 25 13:06:49.451977 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:49.451906    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
May 25 13:06:49.452365 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:49.452250    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
May 25 13:06:49.456876 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:49.456831    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-7779bbdd78-vh2bs"]
May 25 13:06:49.498289 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:49.498254    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rjhw4\" (UniqueName: \"kubernetes.io/projected/72ab7f6c-5589-4ca4-88e4-e975cc094813-kube-api-access-rjhw4\") pod \"prometheus-operator-7779bbdd78-vh2bs\" (UID: \"72ab7f6c-5589-4ca4-88e4-e975cc094813\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-vh2bs"
May 25 13:06:49.498499 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:49.498310    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/72ab7f6c-5589-4ca4-88e4-e975cc094813-prometheus-operator-tls\") pod \"prometheus-operator-7779bbdd78-vh2bs\" (UID: \"72ab7f6c-5589-4ca4-88e4-e975cc094813\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-vh2bs"
May 25 13:06:49.498499 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:49.498336    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/72ab7f6c-5589-4ca4-88e4-e975cc094813-metrics-client-ca\") pod \"prometheus-operator-7779bbdd78-vh2bs\" (UID: \"72ab7f6c-5589-4ca4-88e4-e975cc094813\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-vh2bs"
May 25 13:06:49.498499 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:49.498388    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/72ab7f6c-5589-4ca4-88e4-e975cc094813-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7779bbdd78-vh2bs\" (UID: \"72ab7f6c-5589-4ca4-88e4-e975cc094813\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-vh2bs"
May 25 13:06:49.599702 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:49.599653    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-rjhw4\" (UniqueName: \"kubernetes.io/projected/72ab7f6c-5589-4ca4-88e4-e975cc094813-kube-api-access-rjhw4\") pod \"prometheus-operator-7779bbdd78-vh2bs\" (UID: \"72ab7f6c-5589-4ca4-88e4-e975cc094813\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-vh2bs"
May 25 13:06:49.599890 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:49.599721    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/72ab7f6c-5589-4ca4-88e4-e975cc094813-prometheus-operator-tls\") pod \"prometheus-operator-7779bbdd78-vh2bs\" (UID: \"72ab7f6c-5589-4ca4-88e4-e975cc094813\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-vh2bs"
May 25 13:06:49.599962 ip-10-0-143-235 kubenswrapper[2557]: E0525 13:06:49.599886    2557 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found
May 25 13:06:49.600341 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:49.600311    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/72ab7f6c-5589-4ca4-88e4-e975cc094813-metrics-client-ca\") pod \"prometheus-operator-7779bbdd78-vh2bs\" (UID: \"72ab7f6c-5589-4ca4-88e4-e975cc094813\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-vh2bs"
May 25 13:06:49.600471 ip-10-0-143-235 kubenswrapper[2557]: E0525 13:06:49.600396    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/72ab7f6c-5589-4ca4-88e4-e975cc094813-prometheus-operator-tls podName:72ab7f6c-5589-4ca4-88e4-e975cc094813 nodeName:}" failed. No retries permitted until 2026-05-25 13:06:50.100372086 +0000 UTC m=+62.563388236 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/72ab7f6c-5589-4ca4-88e4-e975cc094813-prometheus-operator-tls") pod "prometheus-operator-7779bbdd78-vh2bs" (UID: "72ab7f6c-5589-4ca4-88e4-e975cc094813") : secret "prometheus-operator-tls" not found
May 25 13:06:49.600471 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:49.600432    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/72ab7f6c-5589-4ca4-88e4-e975cc094813-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7779bbdd78-vh2bs\" (UID: \"72ab7f6c-5589-4ca4-88e4-e975cc094813\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-vh2bs"
May 25 13:06:49.601016 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:49.600995    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/72ab7f6c-5589-4ca4-88e4-e975cc094813-metrics-client-ca\") pod \"prometheus-operator-7779bbdd78-vh2bs\" (UID: \"72ab7f6c-5589-4ca4-88e4-e975cc094813\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-vh2bs"
May 25 13:06:49.603049 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:49.603028    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/72ab7f6c-5589-4ca4-88e4-e975cc094813-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7779bbdd78-vh2bs\" (UID: \"72ab7f6c-5589-4ca4-88e4-e975cc094813\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-vh2bs"
May 25 13:06:49.611430 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:49.611399    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjhw4\" (UniqueName: \"kubernetes.io/projected/72ab7f6c-5589-4ca4-88e4-e975cc094813-kube-api-access-rjhw4\") pod \"prometheus-operator-7779bbdd78-vh2bs\" (UID: \"72ab7f6c-5589-4ca4-88e4-e975cc094813\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-vh2bs"
May 25 13:06:50.105151 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:50.105103    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/72ab7f6c-5589-4ca4-88e4-e975cc094813-prometheus-operator-tls\") pod \"prometheus-operator-7779bbdd78-vh2bs\" (UID: \"72ab7f6c-5589-4ca4-88e4-e975cc094813\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-vh2bs"
May 25 13:06:50.107696 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:50.107672    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/72ab7f6c-5589-4ca4-88e4-e975cc094813-prometheus-operator-tls\") pod \"prometheus-operator-7779bbdd78-vh2bs\" (UID: \"72ab7f6c-5589-4ca4-88e4-e975cc094813\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-vh2bs"
May 25 13:06:50.358464 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:50.358342    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-7779bbdd78-vh2bs"
May 25 13:06:50.473039 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:50.473004    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5594bdbb97-l255d" event={"ID":"4bb7cd7c-97a8-467d-b7d5-e1db2943618f","Type":"ContainerStarted","Data":"8f4ec518d811536ef0b801bcc2902390559fee9158b94b5f682b49862e2487d3"}
May 25 13:06:50.483135 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:50.483100    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-7779bbdd78-vh2bs"]
May 25 13:06:50.487579 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:06:50.487553    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72ab7f6c_5589_4ca4_88e4_e975cc094813.slice/crio-c760a7b961042069b710da596d270b0617c1aba0f92fb0e5b1e0517a51bc0982 WatchSource:0}: Error finding container c760a7b961042069b710da596d270b0617c1aba0f92fb0e5b1e0517a51bc0982: Status 404 returned error can't find the container with id c760a7b961042069b710da596d270b0617c1aba0f92fb0e5b1e0517a51bc0982
May 25 13:06:50.491035 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:50.490996    2557 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-console/console-5594bdbb97-l255d" podStartSLOduration=0.983960188 podStartE2EDuration="3.490980942s" podCreationTimestamp="2026-05-25 13:06:47 +0000 UTC" firstStartedPulling="2026-05-25 13:06:47.650089984 +0000 UTC m=+60.113106119" lastFinishedPulling="2026-05-25 13:06:50.157110735 +0000 UTC m=+62.620126873" observedRunningTime="2026-05-25 13:06:50.490596506 +0000 UTC m=+62.953612662" watchObservedRunningTime="2026-05-25 13:06:50.490980942 +0000 UTC m=+62.953997102"
May 25 13:06:51.477254 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:51.477207    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7779bbdd78-vh2bs" event={"ID":"72ab7f6c-5589-4ca4-88e4-e975cc094813","Type":"ContainerStarted","Data":"c760a7b961042069b710da596d270b0617c1aba0f92fb0e5b1e0517a51bc0982"}
May 25 13:06:52.484293 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:52.484255    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7779bbdd78-vh2bs" event={"ID":"72ab7f6c-5589-4ca4-88e4-e975cc094813","Type":"ContainerStarted","Data":"db37da4a4bdce18a1e05cbf44e8a3208f4ab407ce70e9957bcf0a44301baa716"}
May 25 13:06:52.484293 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:52.484296    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7779bbdd78-vh2bs" event={"ID":"72ab7f6c-5589-4ca4-88e4-e975cc094813","Type":"ContainerStarted","Data":"d7d7c35f08995e4e7941b2a6345d49426e795a19cd0ee8a63b2137098b2710a1"}
May 25 13:06:52.501523 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:52.501465    2557 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-7779bbdd78-vh2bs" podStartSLOduration=2.390937835 podStartE2EDuration="3.501450355s" podCreationTimestamp="2026-05-25 13:06:49 +0000 UTC" firstStartedPulling="2026-05-25 13:06:50.489265603 +0000 UTC m=+62.952281737" lastFinishedPulling="2026-05-25 13:06:51.599778122 +0000 UTC m=+64.062794257" observedRunningTime="2026-05-25 13:06:52.500600979 +0000 UTC m=+64.963617136" watchObservedRunningTime="2026-05-25 13:06:52.501450355 +0000 UTC m=+64.964466511"
May 25 13:06:53.832168 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:53.832127    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c2f1987-8965-449c-b426-b46515332392-metrics-certs\") pod \"network-metrics-daemon-5lm4j\" (UID: \"8c2f1987-8965-449c-b426-b46515332392\") " pod="openshift-multus/network-metrics-daemon-5lm4j"
May 25 13:06:53.834980 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:53.834961    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
May 25 13:06:53.844784 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:53.844757    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c2f1987-8965-449c-b426-b46515332392-metrics-certs\") pod \"network-metrics-daemon-5lm4j\" (UID: \"8c2f1987-8965-449c-b426-b46515332392\") " pod="openshift-multus/network-metrics-daemon-5lm4j"
May 25 13:06:53.932714 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:53.932661    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-4f7rk\" (UniqueName: \"kubernetes.io/projected/a8d6074e-8880-4fd2-b612-de33be9d873e-kube-api-access-4f7rk\") pod \"network-check-target-96mr4\" (UID: \"a8d6074e-8880-4fd2-b612-de33be9d873e\") " pod="openshift-network-diagnostics/network-check-target-96mr4"
May 25 13:06:53.935491 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:53.935472    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
May 25 13:06:53.949133 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:53.949108    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
May 25 13:06:53.957051 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:53.957029    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-clhmx\""
May 25 13:06:53.957221 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:53.957198    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4f7rk\" (UniqueName: \"kubernetes.io/projected/a8d6074e-8880-4fd2-b612-de33be9d873e-kube-api-access-4f7rk\") pod \"network-check-target-96mr4\" (UID: \"a8d6074e-8880-4fd2-b612-de33be9d873e\") " pod="openshift-network-diagnostics/network-check-target-96mr4"
May 25 13:06:53.962807 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:53.962787    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-gcddr\""
May 25 13:06:53.964885 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:53.964865    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5lm4j"
May 25 13:06:53.970582 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:53.970562    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-96mr4"
May 25 13:06:54.095470 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:54.095389    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-5lm4j"]
May 25 13:06:54.098698 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:06:54.098666    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c2f1987_8965_449c_b426_b46515332392.slice/crio-081e965d1e2168265c0d6c1948265b98139cad63f9cf45f30e410c3f57d87b3c WatchSource:0}: Error finding container 081e965d1e2168265c0d6c1948265b98139cad63f9cf45f30e410c3f57d87b3c: Status 404 returned error can't find the container with id 081e965d1e2168265c0d6c1948265b98139cad63f9cf45f30e410c3f57d87b3c
May 25 13:06:54.114801 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:54.114774    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-96mr4"]
May 25 13:06:54.118266 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:06:54.118240    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8d6074e_8880_4fd2_b612_de33be9d873e.slice/crio-86de49450335ce6edf8a6230a7b20017a10f6d3d2c0090acee048ecdb201488f WatchSource:0}: Error finding container 86de49450335ce6edf8a6230a7b20017a10f6d3d2c0090acee048ecdb201488f: Status 404 returned error can't find the container with id 86de49450335ce6edf8a6230a7b20017a10f6d3d2c0090acee048ecdb201488f
May 25 13:06:54.492236 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:54.492135    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-96mr4" event={"ID":"a8d6074e-8880-4fd2-b612-de33be9d873e","Type":"ContainerStarted","Data":"86de49450335ce6edf8a6230a7b20017a10f6d3d2c0090acee048ecdb201488f"}
May 25 13:06:54.493187 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:54.493147    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-5lm4j" event={"ID":"8c2f1987-8965-449c-b426-b46515332392","Type":"ContainerStarted","Data":"081e965d1e2168265c0d6c1948265b98139cad63f9cf45f30e410c3f57d87b3c"}
May 25 13:06:54.834020 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:54.833979    2557 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-9c959559f-klxgf"]
May 25 13:06:54.853867 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:54.853824    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-9c959559f-klxgf"]
May 25 13:06:54.854015 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:54.853986    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-9c959559f-klxgf"
May 25 13:06:54.857094 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:54.856853    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
May 25 13:06:54.857507 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:54.857479    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-bb6h9\""
May 25 13:06:54.857731 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:54.857712    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
May 25 13:06:54.893286 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:54.893246    2557 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-54fd6bc5c4-cns6h"]
May 25 13:06:54.910213 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:54.910172    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-54fd6bc5c4-cns6h"]
May 25 13:06:54.910384 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:54.910310    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-cns6h"
May 25 13:06:54.913450 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:54.913425    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
May 25 13:06:54.913703 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:54.913683    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
May 25 13:06:54.913818 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:54.913737    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-fjft5\""
May 25 13:06:54.913895 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:54.913875    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
May 25 13:06:54.923072 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:54.923048    2557 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-sxvw9"]
May 25 13:06:54.939499 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:54.939465    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1bd034d7-2aef-4292-af7a-9a72e9fa621b-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-9c959559f-klxgf\" (UID: \"1bd034d7-2aef-4292-af7a-9a72e9fa621b\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-klxgf"
May 25 13:06:54.939670 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:54.939515    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44htz\" (UniqueName: \"kubernetes.io/projected/1bd034d7-2aef-4292-af7a-9a72e9fa621b-kube-api-access-44htz\") pod \"openshift-state-metrics-9c959559f-klxgf\" (UID: \"1bd034d7-2aef-4292-af7a-9a72e9fa621b\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-klxgf"
May 25 13:06:54.939670 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:54.939562    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/1bd034d7-2aef-4292-af7a-9a72e9fa621b-openshift-state-metrics-tls\") pod \"openshift-state-metrics-9c959559f-klxgf\" (UID: \"1bd034d7-2aef-4292-af7a-9a72e9fa621b\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-klxgf"
May 25 13:06:54.939756 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:54.939664    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1bd034d7-2aef-4292-af7a-9a72e9fa621b-metrics-client-ca\") pod \"openshift-state-metrics-9c959559f-klxgf\" (UID: \"1bd034d7-2aef-4292-af7a-9a72e9fa621b\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-klxgf"
May 25 13:06:54.946653 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:54.946611    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-sxvw9"
May 25 13:06:54.949704 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:54.949444    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
May 25 13:06:54.949704 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:54.949500    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-47s74\""
May 25 13:06:54.949969 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:54.949753    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
May 25 13:06:54.949969 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:54.949832    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
May 25 13:06:55.040530 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:55.040498    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/f83b731b-63df-4555-95a3-ef82a12a049d-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-54fd6bc5c4-cns6h\" (UID: \"f83b731b-63df-4555-95a3-ef82a12a049d\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-cns6h"
May 25 13:06:55.040711 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:55.040555    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a6e33706-ba69-4aa3-9208-d2b97dfda444-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-sxvw9\" (UID: \"a6e33706-ba69-4aa3-9208-d2b97dfda444\") " pod="openshift-monitoring/node-exporter-sxvw9"
May 25 13:06:55.040711 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:55.040583    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/f83b731b-63df-4555-95a3-ef82a12a049d-volume-directive-shadow\") pod \"kube-state-metrics-54fd6bc5c4-cns6h\" (UID: \"f83b731b-63df-4555-95a3-ef82a12a049d\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-cns6h"
May 25 13:06:55.040711 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:55.040610    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a6e33706-ba69-4aa3-9208-d2b97dfda444-sys\") pod \"node-exporter-sxvw9\" (UID: \"a6e33706-ba69-4aa3-9208-d2b97dfda444\") " pod="openshift-monitoring/node-exporter-sxvw9"
May 25 13:06:55.040711 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:55.040635    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a6e33706-ba69-4aa3-9208-d2b97dfda444-node-exporter-tls\") pod \"node-exporter-sxvw9\" (UID: \"a6e33706-ba69-4aa3-9208-d2b97dfda444\") " pod="openshift-monitoring/node-exporter-sxvw9"
May 25 13:06:55.040711 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:55.040656    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-htgnl\" (UniqueName: \"kubernetes.io/projected/a6e33706-ba69-4aa3-9208-d2b97dfda444-kube-api-access-htgnl\") pod \"node-exporter-sxvw9\" (UID: \"a6e33706-ba69-4aa3-9208-d2b97dfda444\") " pod="openshift-monitoring/node-exporter-sxvw9"
May 25 13:06:55.040711 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:55.040678    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/f83b731b-63df-4555-95a3-ef82a12a049d-kube-state-metrics-tls\") pod \"kube-state-metrics-54fd6bc5c4-cns6h\" (UID: \"f83b731b-63df-4555-95a3-ef82a12a049d\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-cns6h"
May 25 13:06:55.040711 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:55.040708    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1bd034d7-2aef-4292-af7a-9a72e9fa621b-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-9c959559f-klxgf\" (UID: \"1bd034d7-2aef-4292-af7a-9a72e9fa621b\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-klxgf"
May 25 13:06:55.041078 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:55.040746    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a6e33706-ba69-4aa3-9208-d2b97dfda444-node-exporter-accelerators-collector-config\") pod \"node-exporter-sxvw9\" (UID: \"a6e33706-ba69-4aa3-9208-d2b97dfda444\") " pod="openshift-monitoring/node-exporter-sxvw9"
May 25 13:06:55.041078 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:55.040771    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f83b731b-63df-4555-95a3-ef82a12a049d-metrics-client-ca\") pod \"kube-state-metrics-54fd6bc5c4-cns6h\" (UID: \"f83b731b-63df-4555-95a3-ef82a12a049d\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-cns6h"
May 25 13:06:55.041078 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:55.040803    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a6e33706-ba69-4aa3-9208-d2b97dfda444-node-exporter-textfile\") pod \"node-exporter-sxvw9\" (UID: \"a6e33706-ba69-4aa3-9208-d2b97dfda444\") " pod="openshift-monitoring/node-exporter-sxvw9"
May 25 13:06:55.041078 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:55.040831    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a6e33706-ba69-4aa3-9208-d2b97dfda444-node-exporter-wtmp\") pod \"node-exporter-sxvw9\" (UID: \"a6e33706-ba69-4aa3-9208-d2b97dfda444\") " pod="openshift-monitoring/node-exporter-sxvw9"
May 25 13:06:55.041078 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:55.040864    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6km8x\" (UniqueName: \"kubernetes.io/projected/f83b731b-63df-4555-95a3-ef82a12a049d-kube-api-access-6km8x\") pod \"kube-state-metrics-54fd6bc5c4-cns6h\" (UID: \"f83b731b-63df-4555-95a3-ef82a12a049d\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-cns6h"
May 25 13:06:55.041078 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:55.040899    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1bd034d7-2aef-4292-af7a-9a72e9fa621b-metrics-client-ca\") pod \"openshift-state-metrics-9c959559f-klxgf\" (UID: \"1bd034d7-2aef-4292-af7a-9a72e9fa621b\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-klxgf"
May 25 13:06:55.041078 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:55.040944    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-44htz\" (UniqueName: \"kubernetes.io/projected/1bd034d7-2aef-4292-af7a-9a72e9fa621b-kube-api-access-44htz\") pod \"openshift-state-metrics-9c959559f-klxgf\" (UID: \"1bd034d7-2aef-4292-af7a-9a72e9fa621b\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-klxgf"
May 25 13:06:55.041078 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:55.040977    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f83b731b-63df-4555-95a3-ef82a12a049d-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-54fd6bc5c4-cns6h\" (UID: \"f83b731b-63df-4555-95a3-ef82a12a049d\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-cns6h"
May 25 13:06:55.041078 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:55.041008    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a6e33706-ba69-4aa3-9208-d2b97dfda444-root\") pod \"node-exporter-sxvw9\" (UID: \"a6e33706-ba69-4aa3-9208-d2b97dfda444\") " pod="openshift-monitoring/node-exporter-sxvw9"
May 25 13:06:55.041078 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:55.041034    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a6e33706-ba69-4aa3-9208-d2b97dfda444-metrics-client-ca\") pod \"node-exporter-sxvw9\" (UID: \"a6e33706-ba69-4aa3-9208-d2b97dfda444\") " pod="openshift-monitoring/node-exporter-sxvw9"
May 25 13:06:55.041078 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:55.041065    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/1bd034d7-2aef-4292-af7a-9a72e9fa621b-openshift-state-metrics-tls\") pod \"openshift-state-metrics-9c959559f-klxgf\" (UID: \"1bd034d7-2aef-4292-af7a-9a72e9fa621b\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-klxgf"
May 25 13:06:55.042288 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:55.042238    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1bd034d7-2aef-4292-af7a-9a72e9fa621b-metrics-client-ca\") pod \"openshift-state-metrics-9c959559f-klxgf\" (UID: \"1bd034d7-2aef-4292-af7a-9a72e9fa621b\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-klxgf"
May 25 13:06:55.044130 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:55.044091    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1bd034d7-2aef-4292-af7a-9a72e9fa621b-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-9c959559f-klxgf\" (UID: \"1bd034d7-2aef-4292-af7a-9a72e9fa621b\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-klxgf"
May 25 13:06:55.045041 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:55.045015    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/1bd034d7-2aef-4292-af7a-9a72e9fa621b-openshift-state-metrics-tls\") pod \"openshift-state-metrics-9c959559f-klxgf\" (UID: \"1bd034d7-2aef-4292-af7a-9a72e9fa621b\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-klxgf"
May 25 13:06:55.051313 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:55.051275    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-44htz\" (UniqueName: \"kubernetes.io/projected/1bd034d7-2aef-4292-af7a-9a72e9fa621b-kube-api-access-44htz\") pod \"openshift-state-metrics-9c959559f-klxgf\" (UID: \"1bd034d7-2aef-4292-af7a-9a72e9fa621b\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-klxgf"
May 25 13:06:55.141856 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:55.141760    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f83b731b-63df-4555-95a3-ef82a12a049d-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-54fd6bc5c4-cns6h\" (UID: \"f83b731b-63df-4555-95a3-ef82a12a049d\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-cns6h"
May 25 13:06:55.141856 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:55.141817    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a6e33706-ba69-4aa3-9208-d2b97dfda444-root\") pod \"node-exporter-sxvw9\" (UID: \"a6e33706-ba69-4aa3-9208-d2b97dfda444\") " pod="openshift-monitoring/node-exporter-sxvw9"
May 25 13:06:55.141856 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:55.141847    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a6e33706-ba69-4aa3-9208-d2b97dfda444-metrics-client-ca\") pod \"node-exporter-sxvw9\" (UID: \"a6e33706-ba69-4aa3-9208-d2b97dfda444\") " pod="openshift-monitoring/node-exporter-sxvw9"
May 25 13:06:55.142135 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:55.141895    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/f83b731b-63df-4555-95a3-ef82a12a049d-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-54fd6bc5c4-cns6h\" (UID: \"f83b731b-63df-4555-95a3-ef82a12a049d\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-cns6h"
May 25 13:06:55.142135 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:55.141934    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a6e33706-ba69-4aa3-9208-d2b97dfda444-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-sxvw9\" (UID: \"a6e33706-ba69-4aa3-9208-d2b97dfda444\") " pod="openshift-monitoring/node-exporter-sxvw9"
May 25 13:06:55.142135 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:55.141960    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/f83b731b-63df-4555-95a3-ef82a12a049d-volume-directive-shadow\") pod \"kube-state-metrics-54fd6bc5c4-cns6h\" (UID: \"f83b731b-63df-4555-95a3-ef82a12a049d\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-cns6h"
May 25 13:06:55.142135 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:55.141957    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a6e33706-ba69-4aa3-9208-d2b97dfda444-root\") pod \"node-exporter-sxvw9\" (UID: \"a6e33706-ba69-4aa3-9208-d2b97dfda444\") " pod="openshift-monitoring/node-exporter-sxvw9"
May 25 13:06:55.142135 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:55.142022    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a6e33706-ba69-4aa3-9208-d2b97dfda444-sys\") pod \"node-exporter-sxvw9\" (UID: \"a6e33706-ba69-4aa3-9208-d2b97dfda444\") " pod="openshift-monitoring/node-exporter-sxvw9"
May 25 13:06:55.142135 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:55.142056    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a6e33706-ba69-4aa3-9208-d2b97dfda444-node-exporter-tls\") pod \"node-exporter-sxvw9\" (UID: \"a6e33706-ba69-4aa3-9208-d2b97dfda444\") " pod="openshift-monitoring/node-exporter-sxvw9"
May 25 13:06:55.142135 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:55.142082    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-htgnl\" (UniqueName: \"kubernetes.io/projected/a6e33706-ba69-4aa3-9208-d2b97dfda444-kube-api-access-htgnl\") pod \"node-exporter-sxvw9\" (UID: \"a6e33706-ba69-4aa3-9208-d2b97dfda444\") " pod="openshift-monitoring/node-exporter-sxvw9"
May 25 13:06:55.142135 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:55.142109    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/f83b731b-63df-4555-95a3-ef82a12a049d-kube-state-metrics-tls\") pod \"kube-state-metrics-54fd6bc5c4-cns6h\" (UID: \"f83b731b-63df-4555-95a3-ef82a12a049d\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-cns6h"
May 25 13:06:55.142590 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:55.142152    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a6e33706-ba69-4aa3-9208-d2b97dfda444-node-exporter-accelerators-collector-config\") pod \"node-exporter-sxvw9\" (UID: \"a6e33706-ba69-4aa3-9208-d2b97dfda444\") " pod="openshift-monitoring/node-exporter-sxvw9"
May 25 13:06:55.142590 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:55.142181    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f83b731b-63df-4555-95a3-ef82a12a049d-metrics-client-ca\") pod \"kube-state-metrics-54fd6bc5c4-cns6h\" (UID: \"f83b731b-63df-4555-95a3-ef82a12a049d\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-cns6h"
May 25 13:06:55.142590 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:55.142205    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a6e33706-ba69-4aa3-9208-d2b97dfda444-node-exporter-textfile\") pod \"node-exporter-sxvw9\" (UID: \"a6e33706-ba69-4aa3-9208-d2b97dfda444\") " pod="openshift-monitoring/node-exporter-sxvw9"
May 25 13:06:55.142590 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:55.142233    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a6e33706-ba69-4aa3-9208-d2b97dfda444-node-exporter-wtmp\") pod \"node-exporter-sxvw9\" (UID: \"a6e33706-ba69-4aa3-9208-d2b97dfda444\") " pod="openshift-monitoring/node-exporter-sxvw9"
May 25 13:06:55.142590 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:55.142266    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-6km8x\" (UniqueName: \"kubernetes.io/projected/f83b731b-63df-4555-95a3-ef82a12a049d-kube-api-access-6km8x\") pod \"kube-state-metrics-54fd6bc5c4-cns6h\" (UID: \"f83b731b-63df-4555-95a3-ef82a12a049d\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-cns6h"
May 25 13:06:55.142590 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:55.142444    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/f83b731b-63df-4555-95a3-ef82a12a049d-volume-directive-shadow\") pod \"kube-state-metrics-54fd6bc5c4-cns6h\" (UID: \"f83b731b-63df-4555-95a3-ef82a12a049d\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-cns6h"
May 25 13:06:55.142870 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:55.142649    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a6e33706-ba69-4aa3-9208-d2b97dfda444-metrics-client-ca\") pod \"node-exporter-sxvw9\" (UID: \"a6e33706-ba69-4aa3-9208-d2b97dfda444\") " pod="openshift-monitoring/node-exporter-sxvw9"
May 25 13:06:55.142870 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:55.142698    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a6e33706-ba69-4aa3-9208-d2b97dfda444-node-exporter-textfile\") pod \"node-exporter-sxvw9\" (UID: \"a6e33706-ba69-4aa3-9208-d2b97dfda444\") " pod="openshift-monitoring/node-exporter-sxvw9"
May 25 13:06:55.142870 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:55.142737    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a6e33706-ba69-4aa3-9208-d2b97dfda444-node-exporter-wtmp\") pod \"node-exporter-sxvw9\" (UID: \"a6e33706-ba69-4aa3-9208-d2b97dfda444\") " pod="openshift-monitoring/node-exporter-sxvw9"
May 25 13:06:55.143011 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:55.142980    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a6e33706-ba69-4aa3-9208-d2b97dfda444-sys\") pod \"node-exporter-sxvw9\" (UID: \"a6e33706-ba69-4aa3-9208-d2b97dfda444\") " pod="openshift-monitoring/node-exporter-sxvw9"
May 25 13:06:55.143283 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:55.143256    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f83b731b-63df-4555-95a3-ef82a12a049d-metrics-client-ca\") pod \"kube-state-metrics-54fd6bc5c4-cns6h\" (UID: \"f83b731b-63df-4555-95a3-ef82a12a049d\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-cns6h"
May 25 13:06:55.143387 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:55.143313    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a6e33706-ba69-4aa3-9208-d2b97dfda444-node-exporter-accelerators-collector-config\") pod \"node-exporter-sxvw9\" (UID: \"a6e33706-ba69-4aa3-9208-d2b97dfda444\") " pod="openshift-monitoring/node-exporter-sxvw9"
May 25 13:06:55.143560 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:55.143537    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/f83b731b-63df-4555-95a3-ef82a12a049d-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-54fd6bc5c4-cns6h\" (UID: \"f83b731b-63df-4555-95a3-ef82a12a049d\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-cns6h"
May 25 13:06:55.145490 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:55.145436    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a6e33706-ba69-4aa3-9208-d2b97dfda444-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-sxvw9\" (UID: \"a6e33706-ba69-4aa3-9208-d2b97dfda444\") " pod="openshift-monitoring/node-exporter-sxvw9"
May 25 13:06:55.145617 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:55.145545    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/f83b731b-63df-4555-95a3-ef82a12a049d-kube-state-metrics-tls\") pod \"kube-state-metrics-54fd6bc5c4-cns6h\" (UID: \"f83b731b-63df-4555-95a3-ef82a12a049d\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-cns6h"
May 25 13:06:55.145789 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:55.145764    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f83b731b-63df-4555-95a3-ef82a12a049d-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-54fd6bc5c4-cns6h\" (UID: \"f83b731b-63df-4555-95a3-ef82a12a049d\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-cns6h"
May 25 13:06:55.146911 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:55.146856    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a6e33706-ba69-4aa3-9208-d2b97dfda444-node-exporter-tls\") pod \"node-exporter-sxvw9\" (UID: \"a6e33706-ba69-4aa3-9208-d2b97dfda444\") " pod="openshift-monitoring/node-exporter-sxvw9"
May 25 13:06:55.151282 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:55.151257    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-htgnl\" (UniqueName: \"kubernetes.io/projected/a6e33706-ba69-4aa3-9208-d2b97dfda444-kube-api-access-htgnl\") pod \"node-exporter-sxvw9\" (UID: \"a6e33706-ba69-4aa3-9208-d2b97dfda444\") " pod="openshift-monitoring/node-exporter-sxvw9"
May 25 13:06:55.151627 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:55.151603    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6km8x\" (UniqueName: \"kubernetes.io/projected/f83b731b-63df-4555-95a3-ef82a12a049d-kube-api-access-6km8x\") pod \"kube-state-metrics-54fd6bc5c4-cns6h\" (UID: \"f83b731b-63df-4555-95a3-ef82a12a049d\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-cns6h"
May 25 13:06:55.166363 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:55.166312    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-9c959559f-klxgf"
May 25 13:06:55.222495 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:55.222457    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-cns6h"
May 25 13:06:55.258826 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:55.258780    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-sxvw9"
May 25 13:06:55.315008 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:06:55.314974    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6e33706_ba69_4aa3_9208_d2b97dfda444.slice/crio-8277df28b9d5f50c006aa7c64e9e69c42c0f5d3f45c69245797c3263ae008f78 WatchSource:0}: Error finding container 8277df28b9d5f50c006aa7c64e9e69c42c0f5d3f45c69245797c3263ae008f78: Status 404 returned error can't find the container with id 8277df28b9d5f50c006aa7c64e9e69c42c0f5d3f45c69245797c3263ae008f78
May 25 13:06:55.460542 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:55.460506    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-9c959559f-klxgf"]
May 25 13:06:55.465264 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:06:55.465234    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bd034d7_2aef_4292_af7a_9a72e9fa621b.slice/crio-488e615a62863aab399c3878228c8ffb19a834e331f46b87657adeeb43b46d25 WatchSource:0}: Error finding container 488e615a62863aab399c3878228c8ffb19a834e331f46b87657adeeb43b46d25: Status 404 returned error can't find the container with id 488e615a62863aab399c3878228c8ffb19a834e331f46b87657adeeb43b46d25
May 25 13:06:55.481035 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:55.481004    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-54fd6bc5c4-cns6h"]
May 25 13:06:55.484549 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:06:55.484520    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf83b731b_63df_4555_95a3_ef82a12a049d.slice/crio-d168fb9ff79ad2a9cb6fea19013408ff285c8b4be906f61dea5f5c5b5599d3c8 WatchSource:0}: Error finding container d168fb9ff79ad2a9cb6fea19013408ff285c8b4be906f61dea5f5c5b5599d3c8: Status 404 returned error can't find the container with id d168fb9ff79ad2a9cb6fea19013408ff285c8b4be906f61dea5f5c5b5599d3c8
May 25 13:06:55.496987 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:55.496951    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-sxvw9" event={"ID":"a6e33706-ba69-4aa3-9208-d2b97dfda444","Type":"ContainerStarted","Data":"8277df28b9d5f50c006aa7c64e9e69c42c0f5d3f45c69245797c3263ae008f78"}
May 25 13:06:55.498147 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:55.498120    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-cns6h" event={"ID":"f83b731b-63df-4555-95a3-ef82a12a049d","Type":"ContainerStarted","Data":"d168fb9ff79ad2a9cb6fea19013408ff285c8b4be906f61dea5f5c5b5599d3c8"}
May 25 13:06:55.499519 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:55.499492    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-9c959559f-klxgf" event={"ID":"1bd034d7-2aef-4292-af7a-9a72e9fa621b","Type":"ContainerStarted","Data":"488e615a62863aab399c3878228c8ffb19a834e331f46b87657adeeb43b46d25"}
May 25 13:06:55.896073 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:55.896034    2557 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
May 25 13:06:55.899230 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:55.899165    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
May 25 13:06:55.902616 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:55.902308    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
May 25 13:06:55.902616 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:55.902494    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-z5m5x\""
May 25 13:06:55.902616 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:55.902583    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
May 25 13:06:55.902851 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:55.902706    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
May 25 13:06:55.902851 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:55.902826    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
May 25 13:06:55.902851 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:55.902848    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
May 25 13:06:55.903003 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:55.902920    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
May 25 13:06:55.903058 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:55.903027    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
May 25 13:06:55.903058 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:55.903053    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
May 25 13:06:55.903182 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:55.903102    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
May 25 13:06:55.921676 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:55.921647    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
May 25 13:06:55.950430 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:55.950389    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/5f764e8d-9e15-478e-8e6b-50ebf8290b0c-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"5f764e8d-9e15-478e-8e6b-50ebf8290b0c\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 13:06:55.950617 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:55.950443    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5f764e8d-9e15-478e-8e6b-50ebf8290b0c-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"5f764e8d-9e15-478e-8e6b-50ebf8290b0c\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 13:06:55.950617 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:55.950475    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5f764e8d-9e15-478e-8e6b-50ebf8290b0c-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"5f764e8d-9e15-478e-8e6b-50ebf8290b0c\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 13:06:55.950617 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:55.950506    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5f764e8d-9e15-478e-8e6b-50ebf8290b0c-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"5f764e8d-9e15-478e-8e6b-50ebf8290b0c\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 13:06:55.950617 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:55.950553    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/5f764e8d-9e15-478e-8e6b-50ebf8290b0c-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"5f764e8d-9e15-478e-8e6b-50ebf8290b0c\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 13:06:55.950617 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:55.950594    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5f764e8d-9e15-478e-8e6b-50ebf8290b0c-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"5f764e8d-9e15-478e-8e6b-50ebf8290b0c\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 13:06:55.950887 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:55.950624    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/5f764e8d-9e15-478e-8e6b-50ebf8290b0c-config-volume\") pod \"alertmanager-main-0\" (UID: \"5f764e8d-9e15-478e-8e6b-50ebf8290b0c\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 13:06:55.950887 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:55.950649    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5f764e8d-9e15-478e-8e6b-50ebf8290b0c-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"5f764e8d-9e15-478e-8e6b-50ebf8290b0c\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 13:06:55.950887 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:55.950680    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5f764e8d-9e15-478e-8e6b-50ebf8290b0c-config-out\") pod \"alertmanager-main-0\" (UID: \"5f764e8d-9e15-478e-8e6b-50ebf8290b0c\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 13:06:55.950887 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:55.950704    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5f764e8d-9e15-478e-8e6b-50ebf8290b0c-web-config\") pod \"alertmanager-main-0\" (UID: \"5f764e8d-9e15-478e-8e6b-50ebf8290b0c\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 13:06:55.950887 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:55.950729    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5f764e8d-9e15-478e-8e6b-50ebf8290b0c-tls-assets\") pod \"alertmanager-main-0\" (UID: \"5f764e8d-9e15-478e-8e6b-50ebf8290b0c\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 13:06:55.950887 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:55.950837    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/5f764e8d-9e15-478e-8e6b-50ebf8290b0c-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"5f764e8d-9e15-478e-8e6b-50ebf8290b0c\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 13:06:55.950887 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:55.950875    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8rqb\" (UniqueName: \"kubernetes.io/projected/5f764e8d-9e15-478e-8e6b-50ebf8290b0c-kube-api-access-g8rqb\") pod \"alertmanager-main-0\" (UID: \"5f764e8d-9e15-478e-8e6b-50ebf8290b0c\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 13:06:56.053394 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:56.052527    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/5f764e8d-9e15-478e-8e6b-50ebf8290b0c-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"5f764e8d-9e15-478e-8e6b-50ebf8290b0c\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 13:06:56.053394 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:56.052603    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-g8rqb\" (UniqueName: \"kubernetes.io/projected/5f764e8d-9e15-478e-8e6b-50ebf8290b0c-kube-api-access-g8rqb\") pod \"alertmanager-main-0\" (UID: \"5f764e8d-9e15-478e-8e6b-50ebf8290b0c\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 13:06:56.053394 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:56.052654    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/5f764e8d-9e15-478e-8e6b-50ebf8290b0c-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"5f764e8d-9e15-478e-8e6b-50ebf8290b0c\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 13:06:56.053394 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:56.052686    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5f764e8d-9e15-478e-8e6b-50ebf8290b0c-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"5f764e8d-9e15-478e-8e6b-50ebf8290b0c\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 13:06:56.053394 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:56.052711    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5f764e8d-9e15-478e-8e6b-50ebf8290b0c-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"5f764e8d-9e15-478e-8e6b-50ebf8290b0c\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 13:06:56.053394 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:56.052740    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5f764e8d-9e15-478e-8e6b-50ebf8290b0c-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"5f764e8d-9e15-478e-8e6b-50ebf8290b0c\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 13:06:56.053394 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:56.052785    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/5f764e8d-9e15-478e-8e6b-50ebf8290b0c-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"5f764e8d-9e15-478e-8e6b-50ebf8290b0c\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 13:06:56.053394 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:56.052824    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5f764e8d-9e15-478e-8e6b-50ebf8290b0c-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"5f764e8d-9e15-478e-8e6b-50ebf8290b0c\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 13:06:56.053394 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:56.052855    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/5f764e8d-9e15-478e-8e6b-50ebf8290b0c-config-volume\") pod \"alertmanager-main-0\" (UID: \"5f764e8d-9e15-478e-8e6b-50ebf8290b0c\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 13:06:56.053394 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:56.052879    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5f764e8d-9e15-478e-8e6b-50ebf8290b0c-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"5f764e8d-9e15-478e-8e6b-50ebf8290b0c\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 13:06:56.053394 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:56.052911    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5f764e8d-9e15-478e-8e6b-50ebf8290b0c-config-out\") pod \"alertmanager-main-0\" (UID: \"5f764e8d-9e15-478e-8e6b-50ebf8290b0c\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 13:06:56.053394 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:56.052937    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5f764e8d-9e15-478e-8e6b-50ebf8290b0c-web-config\") pod \"alertmanager-main-0\" (UID: \"5f764e8d-9e15-478e-8e6b-50ebf8290b0c\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 13:06:56.053394 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:56.052961    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5f764e8d-9e15-478e-8e6b-50ebf8290b0c-tls-assets\") pod \"alertmanager-main-0\" (UID: \"5f764e8d-9e15-478e-8e6b-50ebf8290b0c\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 13:06:56.053394 ip-10-0-143-235 kubenswrapper[2557]: E0525 13:06:56.053094    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5f764e8d-9e15-478e-8e6b-50ebf8290b0c-alertmanager-trusted-ca-bundle podName:5f764e8d-9e15-478e-8e6b-50ebf8290b0c nodeName:}" failed. No retries permitted until 2026-05-25 13:06:56.55306718 +0000 UTC m=+69.016083340 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "alertmanager-trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/5f764e8d-9e15-478e-8e6b-50ebf8290b0c-alertmanager-trusted-ca-bundle") pod "alertmanager-main-0" (UID: "5f764e8d-9e15-478e-8e6b-50ebf8290b0c") : configmap references non-existent config key: ca-bundle.crt
May 25 13:06:56.054228 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:56.053942    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/5f764e8d-9e15-478e-8e6b-50ebf8290b0c-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"5f764e8d-9e15-478e-8e6b-50ebf8290b0c\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 13:06:56.055681 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:56.055464    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5f764e8d-9e15-478e-8e6b-50ebf8290b0c-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"5f764e8d-9e15-478e-8e6b-50ebf8290b0c\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 13:06:56.057006 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:56.056953    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/5f764e8d-9e15-478e-8e6b-50ebf8290b0c-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"5f764e8d-9e15-478e-8e6b-50ebf8290b0c\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 13:06:56.058571 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:56.058542    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/5f764e8d-9e15-478e-8e6b-50ebf8290b0c-config-volume\") pod \"alertmanager-main-0\" (UID: \"5f764e8d-9e15-478e-8e6b-50ebf8290b0c\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 13:06:56.059610 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:56.059562    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5f764e8d-9e15-478e-8e6b-50ebf8290b0c-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"5f764e8d-9e15-478e-8e6b-50ebf8290b0c\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 13:06:56.060292 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:56.060121    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5f764e8d-9e15-478e-8e6b-50ebf8290b0c-config-out\") pod \"alertmanager-main-0\" (UID: \"5f764e8d-9e15-478e-8e6b-50ebf8290b0c\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 13:06:56.060429 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:56.060406    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5f764e8d-9e15-478e-8e6b-50ebf8290b0c-tls-assets\") pod \"alertmanager-main-0\" (UID: \"5f764e8d-9e15-478e-8e6b-50ebf8290b0c\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 13:06:56.060879 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:56.060854    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5f764e8d-9e15-478e-8e6b-50ebf8290b0c-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"5f764e8d-9e15-478e-8e6b-50ebf8290b0c\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 13:06:56.060979 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:56.060898    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5f764e8d-9e15-478e-8e6b-50ebf8290b0c-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"5f764e8d-9e15-478e-8e6b-50ebf8290b0c\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 13:06:56.062141 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:56.062094    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5f764e8d-9e15-478e-8e6b-50ebf8290b0c-web-config\") pod \"alertmanager-main-0\" (UID: \"5f764e8d-9e15-478e-8e6b-50ebf8290b0c\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 13:06:56.063476 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:56.063362    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/5f764e8d-9e15-478e-8e6b-50ebf8290b0c-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"5f764e8d-9e15-478e-8e6b-50ebf8290b0c\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 13:06:56.067221 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:56.067197    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8rqb\" (UniqueName: \"kubernetes.io/projected/5f764e8d-9e15-478e-8e6b-50ebf8290b0c-kube-api-access-g8rqb\") pod \"alertmanager-main-0\" (UID: \"5f764e8d-9e15-478e-8e6b-50ebf8290b0c\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 13:06:56.462799 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:56.462767    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-5cwgw"
May 25 13:06:56.505078 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:56.505002    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-9c959559f-klxgf" event={"ID":"1bd034d7-2aef-4292-af7a-9a72e9fa621b","Type":"ContainerStarted","Data":"99cd7dbbbd4e272ec67cca17c6468c24f21deb6e33290ce4ccb934767ceba5df"}
May 25 13:06:56.505078 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:56.505047    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-9c959559f-klxgf" event={"ID":"1bd034d7-2aef-4292-af7a-9a72e9fa621b","Type":"ContainerStarted","Data":"eb75de28d2e6eedc4d5bbe1795ba59475105eceb1ad3c5bcb209e48406afa6ec"}
May 25 13:06:56.506923 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:56.506889    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-5lm4j" event={"ID":"8c2f1987-8965-449c-b426-b46515332392","Type":"ContainerStarted","Data":"7a011ebb0bae5e609dfd489dbf00d7e530b423899c8f0b046256c847c1f20559"}
May 25 13:06:56.507049 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:56.506932    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-5lm4j" event={"ID":"8c2f1987-8965-449c-b426-b46515332392","Type":"ContainerStarted","Data":"4ad8b3b2a6aa0c5714249b55ab19833acb97b69a7d8585c8598d52769bb040fa"}
May 25 13:06:56.522977 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:56.522921    2557 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-5lm4j" podStartSLOduration=67.084330583 podStartE2EDuration="1m8.522904658s" podCreationTimestamp="2026-05-25 13:05:48 +0000 UTC" firstStartedPulling="2026-05-25 13:06:54.100555279 +0000 UTC m=+66.563571414" lastFinishedPulling="2026-05-25 13:06:55.539129348 +0000 UTC m=+68.002145489" observedRunningTime="2026-05-25 13:06:56.522366439 +0000 UTC m=+68.985382596" watchObservedRunningTime="2026-05-25 13:06:56.522904658 +0000 UTC m=+68.985920815"
May 25 13:06:56.558874 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:56.558836    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5f764e8d-9e15-478e-8e6b-50ebf8290b0c-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"5f764e8d-9e15-478e-8e6b-50ebf8290b0c\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 13:06:56.559766 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:56.559737    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5f764e8d-9e15-478e-8e6b-50ebf8290b0c-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"5f764e8d-9e15-478e-8e6b-50ebf8290b0c\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 13:06:56.813869 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:56.813828    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
May 25 13:06:56.932368 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:56.932324    2557 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-ddc675b94-jt4g7"]
May 25 13:06:56.935095 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:56.935073    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-ddc675b94-jt4g7"
May 25 13:06:56.938360 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:56.938318    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
May 25 13:06:56.938465 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:56.938367    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
May 25 13:06:56.938547 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:56.938528    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
May 25 13:06:56.938929 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:56.938911    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
May 25 13:06:56.939009 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:56.938964    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-l7lvq\""
May 25 13:06:56.939550 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:56.939528    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-40e7r0g7miknr\""
May 25 13:06:56.939642 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:56.939576    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
May 25 13:06:56.959200 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:56.959172    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-ddc675b94-jt4g7"]
May 25 13:06:57.063698 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:57.063662    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92vbs\" (UniqueName: \"kubernetes.io/projected/1b20cb0e-7386-4292-860f-cae4aa862a94-kube-api-access-92vbs\") pod \"thanos-querier-ddc675b94-jt4g7\" (UID: \"1b20cb0e-7386-4292-860f-cae4aa862a94\") " pod="openshift-monitoring/thanos-querier-ddc675b94-jt4g7"
May 25 13:06:57.063915 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:57.063716    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/1b20cb0e-7386-4292-860f-cae4aa862a94-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-ddc675b94-jt4g7\" (UID: \"1b20cb0e-7386-4292-860f-cae4aa862a94\") " pod="openshift-monitoring/thanos-querier-ddc675b94-jt4g7"
May 25 13:06:57.063915 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:57.063840    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/1b20cb0e-7386-4292-860f-cae4aa862a94-secret-thanos-querier-tls\") pod \"thanos-querier-ddc675b94-jt4g7\" (UID: \"1b20cb0e-7386-4292-860f-cae4aa862a94\") " pod="openshift-monitoring/thanos-querier-ddc675b94-jt4g7"
May 25 13:06:57.063915 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:57.063877    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1b20cb0e-7386-4292-860f-cae4aa862a94-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-ddc675b94-jt4g7\" (UID: \"1b20cb0e-7386-4292-860f-cae4aa862a94\") " pod="openshift-monitoring/thanos-querier-ddc675b94-jt4g7"
May 25 13:06:57.064068 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:57.063985    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/1b20cb0e-7386-4292-860f-cae4aa862a94-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-ddc675b94-jt4g7\" (UID: \"1b20cb0e-7386-4292-860f-cae4aa862a94\") " pod="openshift-monitoring/thanos-querier-ddc675b94-jt4g7"
May 25 13:06:57.064068 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:57.064053    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1b20cb0e-7386-4292-860f-cae4aa862a94-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-ddc675b94-jt4g7\" (UID: \"1b20cb0e-7386-4292-860f-cae4aa862a94\") " pod="openshift-monitoring/thanos-querier-ddc675b94-jt4g7"
May 25 13:06:57.064151 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:57.064090    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/1b20cb0e-7386-4292-860f-cae4aa862a94-secret-grpc-tls\") pod \"thanos-querier-ddc675b94-jt4g7\" (UID: \"1b20cb0e-7386-4292-860f-cae4aa862a94\") " pod="openshift-monitoring/thanos-querier-ddc675b94-jt4g7"
May 25 13:06:57.064151 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:57.064125    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1b20cb0e-7386-4292-860f-cae4aa862a94-metrics-client-ca\") pod \"thanos-querier-ddc675b94-jt4g7\" (UID: \"1b20cb0e-7386-4292-860f-cae4aa862a94\") " pod="openshift-monitoring/thanos-querier-ddc675b94-jt4g7"
May 25 13:06:57.165501 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:57.165458    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1b20cb0e-7386-4292-860f-cae4aa862a94-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-ddc675b94-jt4g7\" (UID: \"1b20cb0e-7386-4292-860f-cae4aa862a94\") " pod="openshift-monitoring/thanos-querier-ddc675b94-jt4g7"
May 25 13:06:57.165697 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:57.165535    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/1b20cb0e-7386-4292-860f-cae4aa862a94-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-ddc675b94-jt4g7\" (UID: \"1b20cb0e-7386-4292-860f-cae4aa862a94\") " pod="openshift-monitoring/thanos-querier-ddc675b94-jt4g7"
May 25 13:06:57.165697 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:57.165584    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1b20cb0e-7386-4292-860f-cae4aa862a94-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-ddc675b94-jt4g7\" (UID: \"1b20cb0e-7386-4292-860f-cae4aa862a94\") " pod="openshift-monitoring/thanos-querier-ddc675b94-jt4g7"
May 25 13:06:57.165811 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:57.165713    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/1b20cb0e-7386-4292-860f-cae4aa862a94-secret-grpc-tls\") pod \"thanos-querier-ddc675b94-jt4g7\" (UID: \"1b20cb0e-7386-4292-860f-cae4aa862a94\") " pod="openshift-monitoring/thanos-querier-ddc675b94-jt4g7"
May 25 13:06:57.165811 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:57.165761    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1b20cb0e-7386-4292-860f-cae4aa862a94-metrics-client-ca\") pod \"thanos-querier-ddc675b94-jt4g7\" (UID: \"1b20cb0e-7386-4292-860f-cae4aa862a94\") " pod="openshift-monitoring/thanos-querier-ddc675b94-jt4g7"
May 25 13:06:57.165811 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:57.165796    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-92vbs\" (UniqueName: \"kubernetes.io/projected/1b20cb0e-7386-4292-860f-cae4aa862a94-kube-api-access-92vbs\") pod \"thanos-querier-ddc675b94-jt4g7\" (UID: \"1b20cb0e-7386-4292-860f-cae4aa862a94\") " pod="openshift-monitoring/thanos-querier-ddc675b94-jt4g7"
May 25 13:06:57.165955 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:57.165832    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/1b20cb0e-7386-4292-860f-cae4aa862a94-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-ddc675b94-jt4g7\" (UID: \"1b20cb0e-7386-4292-860f-cae4aa862a94\") " pod="openshift-monitoring/thanos-querier-ddc675b94-jt4g7"
May 25 13:06:57.165955 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:57.165885    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/1b20cb0e-7386-4292-860f-cae4aa862a94-secret-thanos-querier-tls\") pod \"thanos-querier-ddc675b94-jt4g7\" (UID: \"1b20cb0e-7386-4292-860f-cae4aa862a94\") " pod="openshift-monitoring/thanos-querier-ddc675b94-jt4g7"
May 25 13:06:57.166665 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:57.166611    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1b20cb0e-7386-4292-860f-cae4aa862a94-metrics-client-ca\") pod \"thanos-querier-ddc675b94-jt4g7\" (UID: \"1b20cb0e-7386-4292-860f-cae4aa862a94\") " pod="openshift-monitoring/thanos-querier-ddc675b94-jt4g7"
May 25 13:06:57.168655 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:57.168627    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/1b20cb0e-7386-4292-860f-cae4aa862a94-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-ddc675b94-jt4g7\" (UID: \"1b20cb0e-7386-4292-860f-cae4aa862a94\") " pod="openshift-monitoring/thanos-querier-ddc675b94-jt4g7"
May 25 13:06:57.168765 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:57.168716    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/1b20cb0e-7386-4292-860f-cae4aa862a94-secret-thanos-querier-tls\") pod \"thanos-querier-ddc675b94-jt4g7\" (UID: \"1b20cb0e-7386-4292-860f-cae4aa862a94\") " pod="openshift-monitoring/thanos-querier-ddc675b94-jt4g7"
May 25 13:06:57.168950 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:57.168933    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1b20cb0e-7386-4292-860f-cae4aa862a94-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-ddc675b94-jt4g7\" (UID: \"1b20cb0e-7386-4292-860f-cae4aa862a94\") " pod="openshift-monitoring/thanos-querier-ddc675b94-jt4g7"
May 25 13:06:57.169008 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:57.168961    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1b20cb0e-7386-4292-860f-cae4aa862a94-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-ddc675b94-jt4g7\" (UID: \"1b20cb0e-7386-4292-860f-cae4aa862a94\") " pod="openshift-monitoring/thanos-querier-ddc675b94-jt4g7"
May 25 13:06:57.169204 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:57.169184    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/1b20cb0e-7386-4292-860f-cae4aa862a94-secret-grpc-tls\") pod \"thanos-querier-ddc675b94-jt4g7\" (UID: \"1b20cb0e-7386-4292-860f-cae4aa862a94\") " pod="openshift-monitoring/thanos-querier-ddc675b94-jt4g7"
May 25 13:06:57.169340 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:57.169315    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/1b20cb0e-7386-4292-860f-cae4aa862a94-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-ddc675b94-jt4g7\" (UID: \"1b20cb0e-7386-4292-860f-cae4aa862a94\") " pod="openshift-monitoring/thanos-querier-ddc675b94-jt4g7"
May 25 13:06:57.175201 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:57.175172    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-92vbs\" (UniqueName: \"kubernetes.io/projected/1b20cb0e-7386-4292-860f-cae4aa862a94-kube-api-access-92vbs\") pod \"thanos-querier-ddc675b94-jt4g7\" (UID: \"1b20cb0e-7386-4292-860f-cae4aa862a94\") " pod="openshift-monitoring/thanos-querier-ddc675b94-jt4g7"
May 25 13:06:57.246323 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:57.246285    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-ddc675b94-jt4g7"
May 25 13:06:57.520586 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:57.520496    2557 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5594bdbb97-l255d"
May 25 13:06:57.520586 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:57.520555    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5594bdbb97-l255d"
May 25 13:06:57.525925 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:57.525899    2557 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5594bdbb97-l255d"
May 25 13:06:58.020254 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:58.019988    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
May 25 13:06:58.022464 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:06:58.022373    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f764e8d_9e15_478e_8e6b_50ebf8290b0c.slice/crio-3824a0699d139c9d513f967f97ea7a4f79df882d6d62bc4d0c0c73b472d12492 WatchSource:0}: Error finding container 3824a0699d139c9d513f967f97ea7a4f79df882d6d62bc4d0c0c73b472d12492: Status 404 returned error can't find the container with id 3824a0699d139c9d513f967f97ea7a4f79df882d6d62bc4d0c0c73b472d12492
May 25 13:06:58.039302 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:58.036560    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-ddc675b94-jt4g7"]
May 25 13:06:58.045859 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:06:58.045811    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b20cb0e_7386_4292_860f_cae4aa862a94.slice/crio-2ba3be4146b89cd5f2464fd283c9d98279d7cc5f0527c55b2dbf34536089a47f WatchSource:0}: Error finding container 2ba3be4146b89cd5f2464fd283c9d98279d7cc5f0527c55b2dbf34536089a47f: Status 404 returned error can't find the container with id 2ba3be4146b89cd5f2464fd283c9d98279d7cc5f0527c55b2dbf34536089a47f
May 25 13:06:58.515418 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:58.515381    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5f764e8d-9e15-478e-8e6b-50ebf8290b0c","Type":"ContainerStarted","Data":"3824a0699d139c9d513f967f97ea7a4f79df882d6d62bc4d0c0c73b472d12492"}
May 25 13:06:58.517668 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:58.517632    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-cns6h" event={"ID":"f83b731b-63df-4555-95a3-ef82a12a049d","Type":"ContainerStarted","Data":"dc1bac0f356e680cbec0d884a1a30001cf2d454f46a5a71f4affd69f3116e2c7"}
May 25 13:06:58.517795 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:58.517676    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-cns6h" event={"ID":"f83b731b-63df-4555-95a3-ef82a12a049d","Type":"ContainerStarted","Data":"f1aa6d539fd687bb3b57fed915ca0a23f73e188888bd58497bc8dbfcad5079f4"}
May 25 13:06:58.517795 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:58.517694    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-cns6h" event={"ID":"f83b731b-63df-4555-95a3-ef82a12a049d","Type":"ContainerStarted","Data":"56a1dfb01094c373d5b0d2699f5dbad518734f2ad22e6da1670605ccbfe079f1"}
May 25 13:06:58.519120 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:58.519095    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-96mr4" event={"ID":"a8d6074e-8880-4fd2-b612-de33be9d873e","Type":"ContainerStarted","Data":"34ca5400f69d5328e630bd9b42f0bad3492c803dfd0a7988b518c786e22ccded"}
May 25 13:06:58.519255 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:58.519234    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-96mr4"
May 25 13:06:58.521008 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:58.520986    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-9c959559f-klxgf" event={"ID":"1bd034d7-2aef-4292-af7a-9a72e9fa621b","Type":"ContainerStarted","Data":"1671c30df4f2ccfcf2eb4812aa16ff890e98723512d0a15c0d0c8de3a53d0320"}
May 25 13:06:58.522291 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:58.522269    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-ddc675b94-jt4g7" event={"ID":"1b20cb0e-7386-4292-860f-cae4aa862a94","Type":"ContainerStarted","Data":"2ba3be4146b89cd5f2464fd283c9d98279d7cc5f0527c55b2dbf34536089a47f"}
May 25 13:06:58.523795 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:58.523770    2557 generic.go:358] "Generic (PLEG): container finished" podID="a6e33706-ba69-4aa3-9208-d2b97dfda444" containerID="da1b8101472aa2b03de8362bac9fb11101ef33abbae267b61d4283d3fe246a4d" exitCode=0
May 25 13:06:58.523904 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:58.523807    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-sxvw9" event={"ID":"a6e33706-ba69-4aa3-9208-d2b97dfda444","Type":"ContainerDied","Data":"da1b8101472aa2b03de8362bac9fb11101ef33abbae267b61d4283d3fe246a4d"}
May 25 13:06:58.529080 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:58.529053    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5594bdbb97-l255d"
May 25 13:06:58.541292 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:58.541236    2557 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-cns6h" podStartSLOduration=2.2593010749999998 podStartE2EDuration="4.54121687s" podCreationTimestamp="2026-05-25 13:06:54 +0000 UTC" firstStartedPulling="2026-05-25 13:06:55.533518152 +0000 UTC m=+67.996534287" lastFinishedPulling="2026-05-25 13:06:57.815433942 +0000 UTC m=+70.278450082" observedRunningTime="2026-05-25 13:06:58.539223063 +0000 UTC m=+71.002239221" watchObservedRunningTime="2026-05-25 13:06:58.54121687 +0000 UTC m=+71.004233028"
May 25 13:06:58.599306 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:58.598688    2557 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-96mr4" podStartSLOduration=66.903268925 podStartE2EDuration="1m10.598669189s" podCreationTimestamp="2026-05-25 13:05:48 +0000 UTC" firstStartedPulling="2026-05-25 13:06:54.120122293 +0000 UTC m=+66.583138428" lastFinishedPulling="2026-05-25 13:06:57.815522549 +0000 UTC m=+70.278538692" observedRunningTime="2026-05-25 13:06:58.597292587 +0000 UTC m=+71.060308744" watchObservedRunningTime="2026-05-25 13:06:58.598669189 +0000 UTC m=+71.061685350"
May 25 13:06:58.621539 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:58.620952    2557 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-9c959559f-klxgf" podStartSLOduration=2.471835457 podStartE2EDuration="4.620936446s" podCreationTimestamp="2026-05-25 13:06:54 +0000 UTC" firstStartedPulling="2026-05-25 13:06:55.665801445 +0000 UTC m=+68.128817595" lastFinishedPulling="2026-05-25 13:06:57.814902435 +0000 UTC m=+70.277918584" observedRunningTime="2026-05-25 13:06:58.620147691 +0000 UTC m=+71.083163850" watchObservedRunningTime="2026-05-25 13:06:58.620936446 +0000 UTC m=+71.083952603"
May 25 13:06:59.318394 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:59.318358    2557 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-759cd6b97c-kmz7z"]
May 25 13:06:59.321643 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:59.321610    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-759cd6b97c-kmz7z"
May 25 13:06:59.325854 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:59.325662    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-13pvu3vcl6q7s\""
May 25 13:06:59.325854 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:59.325687    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
May 25 13:06:59.325854 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:59.325724    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-jzgnw\""
May 25 13:06:59.325854 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:59.325786    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
May 25 13:06:59.326108 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:59.326026    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
May 25 13:06:59.327331 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:59.327311    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
May 25 13:06:59.331783 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:59.331760    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-759cd6b97c-kmz7z"]
May 25 13:06:59.386686 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:59.386584    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/81e544fc-5a66-44c8-8301-0941bedeebe5-secret-metrics-server-client-certs\") pod \"metrics-server-759cd6b97c-kmz7z\" (UID: \"81e544fc-5a66-44c8-8301-0941bedeebe5\") " pod="openshift-monitoring/metrics-server-759cd6b97c-kmz7z"
May 25 13:06:59.386686 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:59.386636    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/81e544fc-5a66-44c8-8301-0941bedeebe5-metrics-server-audit-profiles\") pod \"metrics-server-759cd6b97c-kmz7z\" (UID: \"81e544fc-5a66-44c8-8301-0941bedeebe5\") " pod="openshift-monitoring/metrics-server-759cd6b97c-kmz7z"
May 25 13:06:59.386686 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:59.386662    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/81e544fc-5a66-44c8-8301-0941bedeebe5-client-ca-bundle\") pod \"metrics-server-759cd6b97c-kmz7z\" (UID: \"81e544fc-5a66-44c8-8301-0941bedeebe5\") " pod="openshift-monitoring/metrics-server-759cd6b97c-kmz7z"
May 25 13:06:59.386946 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:59.386736    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rz27n\" (UniqueName: \"kubernetes.io/projected/81e544fc-5a66-44c8-8301-0941bedeebe5-kube-api-access-rz27n\") pod \"metrics-server-759cd6b97c-kmz7z\" (UID: \"81e544fc-5a66-44c8-8301-0941bedeebe5\") " pod="openshift-monitoring/metrics-server-759cd6b97c-kmz7z"
May 25 13:06:59.386946 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:59.386776    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/81e544fc-5a66-44c8-8301-0941bedeebe5-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-759cd6b97c-kmz7z\" (UID: \"81e544fc-5a66-44c8-8301-0941bedeebe5\") " pod="openshift-monitoring/metrics-server-759cd6b97c-kmz7z"
May 25 13:06:59.386946 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:59.386819    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/81e544fc-5a66-44c8-8301-0941bedeebe5-audit-log\") pod \"metrics-server-759cd6b97c-kmz7z\" (UID: \"81e544fc-5a66-44c8-8301-0941bedeebe5\") " pod="openshift-monitoring/metrics-server-759cd6b97c-kmz7z"
May 25 13:06:59.386946 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:59.386853    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/81e544fc-5a66-44c8-8301-0941bedeebe5-secret-metrics-server-tls\") pod \"metrics-server-759cd6b97c-kmz7z\" (UID: \"81e544fc-5a66-44c8-8301-0941bedeebe5\") " pod="openshift-monitoring/metrics-server-759cd6b97c-kmz7z"
May 25 13:06:59.488176 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:59.488138    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/81e544fc-5a66-44c8-8301-0941bedeebe5-secret-metrics-server-client-certs\") pod \"metrics-server-759cd6b97c-kmz7z\" (UID: \"81e544fc-5a66-44c8-8301-0941bedeebe5\") " pod="openshift-monitoring/metrics-server-759cd6b97c-kmz7z"
May 25 13:06:59.488176 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:59.488185    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/81e544fc-5a66-44c8-8301-0941bedeebe5-metrics-server-audit-profiles\") pod \"metrics-server-759cd6b97c-kmz7z\" (UID: \"81e544fc-5a66-44c8-8301-0941bedeebe5\") " pod="openshift-monitoring/metrics-server-759cd6b97c-kmz7z"
May 25 13:06:59.488396 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:59.488223    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/81e544fc-5a66-44c8-8301-0941bedeebe5-client-ca-bundle\") pod \"metrics-server-759cd6b97c-kmz7z\" (UID: \"81e544fc-5a66-44c8-8301-0941bedeebe5\") " pod="openshift-monitoring/metrics-server-759cd6b97c-kmz7z"
May 25 13:06:59.488396 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:59.488247    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-rz27n\" (UniqueName: \"kubernetes.io/projected/81e544fc-5a66-44c8-8301-0941bedeebe5-kube-api-access-rz27n\") pod \"metrics-server-759cd6b97c-kmz7z\" (UID: \"81e544fc-5a66-44c8-8301-0941bedeebe5\") " pod="openshift-monitoring/metrics-server-759cd6b97c-kmz7z"
May 25 13:06:59.488396 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:59.488268    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/81e544fc-5a66-44c8-8301-0941bedeebe5-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-759cd6b97c-kmz7z\" (UID: \"81e544fc-5a66-44c8-8301-0941bedeebe5\") " pod="openshift-monitoring/metrics-server-759cd6b97c-kmz7z"
May 25 13:06:59.488396 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:59.488305    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/81e544fc-5a66-44c8-8301-0941bedeebe5-audit-log\") pod \"metrics-server-759cd6b97c-kmz7z\" (UID: \"81e544fc-5a66-44c8-8301-0941bedeebe5\") " pod="openshift-monitoring/metrics-server-759cd6b97c-kmz7z"
May 25 13:06:59.488714 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:59.488515    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/81e544fc-5a66-44c8-8301-0941bedeebe5-secret-metrics-server-tls\") pod \"metrics-server-759cd6b97c-kmz7z\" (UID: \"81e544fc-5a66-44c8-8301-0941bedeebe5\") " pod="openshift-monitoring/metrics-server-759cd6b97c-kmz7z"
May 25 13:06:59.488861 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:59.488839    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/81e544fc-5a66-44c8-8301-0941bedeebe5-audit-log\") pod \"metrics-server-759cd6b97c-kmz7z\" (UID: \"81e544fc-5a66-44c8-8301-0941bedeebe5\") " pod="openshift-monitoring/metrics-server-759cd6b97c-kmz7z"
May 25 13:06:59.489226 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:59.489177    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/81e544fc-5a66-44c8-8301-0941bedeebe5-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-759cd6b97c-kmz7z\" (UID: \"81e544fc-5a66-44c8-8301-0941bedeebe5\") " pod="openshift-monitoring/metrics-server-759cd6b97c-kmz7z"
May 25 13:06:59.489676 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:59.489651    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/81e544fc-5a66-44c8-8301-0941bedeebe5-metrics-server-audit-profiles\") pod \"metrics-server-759cd6b97c-kmz7z\" (UID: \"81e544fc-5a66-44c8-8301-0941bedeebe5\") " pod="openshift-monitoring/metrics-server-759cd6b97c-kmz7z"
May 25 13:06:59.491256 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:59.491213    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/81e544fc-5a66-44c8-8301-0941bedeebe5-secret-metrics-server-client-certs\") pod \"metrics-server-759cd6b97c-kmz7z\" (UID: \"81e544fc-5a66-44c8-8301-0941bedeebe5\") " pod="openshift-monitoring/metrics-server-759cd6b97c-kmz7z"
May 25 13:06:59.491449 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:59.491421    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/81e544fc-5a66-44c8-8301-0941bedeebe5-secret-metrics-server-tls\") pod \"metrics-server-759cd6b97c-kmz7z\" (UID: \"81e544fc-5a66-44c8-8301-0941bedeebe5\") " pod="openshift-monitoring/metrics-server-759cd6b97c-kmz7z"
May 25 13:06:59.491773 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:59.491753    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/81e544fc-5a66-44c8-8301-0941bedeebe5-client-ca-bundle\") pod \"metrics-server-759cd6b97c-kmz7z\" (UID: \"81e544fc-5a66-44c8-8301-0941bedeebe5\") " pod="openshift-monitoring/metrics-server-759cd6b97c-kmz7z"
May 25 13:06:59.498137 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:59.498112    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rz27n\" (UniqueName: \"kubernetes.io/projected/81e544fc-5a66-44c8-8301-0941bedeebe5-kube-api-access-rz27n\") pod \"metrics-server-759cd6b97c-kmz7z\" (UID: \"81e544fc-5a66-44c8-8301-0941bedeebe5\") " pod="openshift-monitoring/metrics-server-759cd6b97c-kmz7z"
May 25 13:06:59.528521 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:59.528490    2557 generic.go:358] "Generic (PLEG): container finished" podID="5f764e8d-9e15-478e-8e6b-50ebf8290b0c" containerID="227f4b96d784eda8bbde0b59b71bfba433233dd9d034b513a0efcd110e20c66e" exitCode=0
May 25 13:06:59.528677 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:59.528579    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5f764e8d-9e15-478e-8e6b-50ebf8290b0c","Type":"ContainerDied","Data":"227f4b96d784eda8bbde0b59b71bfba433233dd9d034b513a0efcd110e20c66e"}
May 25 13:06:59.531299 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:59.531267    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-sxvw9" event={"ID":"a6e33706-ba69-4aa3-9208-d2b97dfda444","Type":"ContainerStarted","Data":"9c68771a7c2cbaa70eb4374a06130ee418bead83d828c2dc67227fd056dcf944"}
May 25 13:06:59.531431 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:59.531312    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-sxvw9" event={"ID":"a6e33706-ba69-4aa3-9208-d2b97dfda444","Type":"ContainerStarted","Data":"d2253d1fab29188df8c7069062ff5348f073323891ff3bdc6fa69bf8c7e03c33"}
May 25 13:06:59.581552 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:59.581502    2557 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-sxvw9" podStartSLOduration=3.083005744 podStartE2EDuration="5.581479904s" podCreationTimestamp="2026-05-25 13:06:54 +0000 UTC" firstStartedPulling="2026-05-25 13:06:55.317081157 +0000 UTC m=+67.780097296" lastFinishedPulling="2026-05-25 13:06:57.815555314 +0000 UTC m=+70.278571456" observedRunningTime="2026-05-25 13:06:59.580188681 +0000 UTC m=+72.043204839" watchObservedRunningTime="2026-05-25 13:06:59.581479904 +0000 UTC m=+72.044496064"
May 25 13:06:59.602571 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:59.602530    2557 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-665c9c8584-7s92b"]
May 25 13:06:59.607091 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:59.607065    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-665c9c8584-7s92b"
May 25 13:06:59.612187 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:59.612163    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"default-dockercfg-8dt4v\""
May 25 13:06:59.612831 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:59.612279    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
May 25 13:06:59.614281 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:59.614225    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-665c9c8584-7s92b"]
May 25 13:06:59.634104 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:59.634044    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-759cd6b97c-kmz7z"
May 25 13:06:59.693303 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:59.693214    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5db702e4-41d1-42ec-b61d-f71b1cb2fe05-monitoring-plugin-cert\") pod \"monitoring-plugin-665c9c8584-7s92b\" (UID: \"5db702e4-41d1-42ec-b61d-f71b1cb2fe05\") " pod="openshift-monitoring/monitoring-plugin-665c9c8584-7s92b"
May 25 13:06:59.794539 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:06:59.794507    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5db702e4-41d1-42ec-b61d-f71b1cb2fe05-monitoring-plugin-cert\") pod \"monitoring-plugin-665c9c8584-7s92b\" (UID: \"5db702e4-41d1-42ec-b61d-f71b1cb2fe05\") " pod="openshift-monitoring/monitoring-plugin-665c9c8584-7s92b"
May 25 13:06:59.794718 ip-10-0-143-235 kubenswrapper[2557]: E0525 13:06:59.794673    2557 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found
May 25 13:06:59.794767 ip-10-0-143-235 kubenswrapper[2557]: E0525 13:06:59.794750    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5db702e4-41d1-42ec-b61d-f71b1cb2fe05-monitoring-plugin-cert podName:5db702e4-41d1-42ec-b61d-f71b1cb2fe05 nodeName:}" failed. No retries permitted until 2026-05-25 13:07:00.294728347 +0000 UTC m=+72.757744486 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/5db702e4-41d1-42ec-b61d-f71b1cb2fe05-monitoring-plugin-cert") pod "monitoring-plugin-665c9c8584-7s92b" (UID: "5db702e4-41d1-42ec-b61d-f71b1cb2fe05") : secret "monitoring-plugin-cert" not found
May 25 13:07:00.174763 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:07:00.174699    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-759cd6b97c-kmz7z"]
May 25 13:07:00.176712 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:07:00.176681    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81e544fc_5a66_44c8_8301_0941bedeebe5.slice/crio-21b575409af33cfa4abb7ca9495c07d309883d4aa33c67475468d2135776346d WatchSource:0}: Error finding container 21b575409af33cfa4abb7ca9495c07d309883d4aa33c67475468d2135776346d: Status 404 returned error can't find the container with id 21b575409af33cfa4abb7ca9495c07d309883d4aa33c67475468d2135776346d
May 25 13:07:00.298748 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:07:00.298714    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5db702e4-41d1-42ec-b61d-f71b1cb2fe05-monitoring-plugin-cert\") pod \"monitoring-plugin-665c9c8584-7s92b\" (UID: \"5db702e4-41d1-42ec-b61d-f71b1cb2fe05\") " pod="openshift-monitoring/monitoring-plugin-665c9c8584-7s92b"
May 25 13:07:00.301402 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:07:00.301322    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5db702e4-41d1-42ec-b61d-f71b1cb2fe05-monitoring-plugin-cert\") pod \"monitoring-plugin-665c9c8584-7s92b\" (UID: \"5db702e4-41d1-42ec-b61d-f71b1cb2fe05\") " pod="openshift-monitoring/monitoring-plugin-665c9c8584-7s92b"
May 25 13:07:00.520015 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:07:00.519977    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-665c9c8584-7s92b"
May 25 13:07:00.537391 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:07:00.537342    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-ddc675b94-jt4g7" event={"ID":"1b20cb0e-7386-4292-860f-cae4aa862a94","Type":"ContainerStarted","Data":"8040edad33eb4d0ee496d34688e1c9fa48687721125d9f77ab4d218d5389a238"}
May 25 13:07:00.537391 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:07:00.537396    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-ddc675b94-jt4g7" event={"ID":"1b20cb0e-7386-4292-860f-cae4aa862a94","Type":"ContainerStarted","Data":"fb0a139595ea6f109ecec42da08b6bc9adaefdd53972449dba26db4f80396991"}
May 25 13:07:00.537557 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:07:00.537412    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-ddc675b94-jt4g7" event={"ID":"1b20cb0e-7386-4292-860f-cae4aa862a94","Type":"ContainerStarted","Data":"a20ed6034c1afa9959b9987f3123fca0239a90572ff9445bf63cdc9ae37c531a"}
May 25 13:07:00.538581 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:07:00.538551    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-759cd6b97c-kmz7z" event={"ID":"81e544fc-5a66-44c8-8301-0941bedeebe5","Type":"ContainerStarted","Data":"21b575409af33cfa4abb7ca9495c07d309883d4aa33c67475468d2135776346d"}
May 25 13:07:00.689330 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:07:00.689278    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-665c9c8584-7s92b"]
May 25 13:07:01.149269 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:07:01.148871    2557 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
May 25 13:07:01.153487 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:07:01.153449    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
May 25 13:07:01.156764 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:07:01.156737    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
May 25 13:07:01.156978 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:07:01.156882    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
May 25 13:07:01.156978 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:07:01.156904    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
May 25 13:07:01.159550 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:07:01.159528    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
May 25 13:07:01.159662 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:07:01.159585    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
May 25 13:07:01.159737 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:07:01.159535    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
May 25 13:07:01.159830 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:07:01.159814    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-b6q7g6ganpbvm\""
May 25 13:07:01.159937 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:07:01.159922    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
May 25 13:07:01.160053 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:07:01.160013    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-4hh6h\""
May 25 13:07:01.163242 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:07:01.162400    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
May 25 13:07:01.163242 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:07:01.162674    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
May 25 13:07:01.163242 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:07:01.162884    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
May 25 13:07:01.163242 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:07:01.163103    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
May 25 13:07:01.169432 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:07:01.169231    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
May 25 13:07:01.173574 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:07:01.170398    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
May 25 13:07:01.256766 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:07:01.256724    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5db702e4_41d1_42ec_b61d_f71b1cb2fe05.slice/crio-f03b07f15232ae6a1665db80f396c4713bf4ee85cf44d9ea3e2f0940ec812972 WatchSource:0}: Error finding container f03b07f15232ae6a1665db80f396c4713bf4ee85cf44d9ea3e2f0940ec812972: Status 404 returned error can't find the container with id f03b07f15232ae6a1665db80f396c4713bf4ee85cf44d9ea3e2f0940ec812972
May 25 13:07:01.311766 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:07:01.311732    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5c8ede39-a1d6-41f5-9576-3fd866ec551d-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"5c8ede39-a1d6-41f5-9576-3fd866ec551d\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 13:07:01.311937 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:07:01.311775    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/5c8ede39-a1d6-41f5-9576-3fd866ec551d-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"5c8ede39-a1d6-41f5-9576-3fd866ec551d\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 13:07:01.311937 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:07:01.311840    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5c8ede39-a1d6-41f5-9576-3fd866ec551d-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"5c8ede39-a1d6-41f5-9576-3fd866ec551d\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 13:07:01.311937 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:07:01.311899    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5c8ede39-a1d6-41f5-9576-3fd866ec551d-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"5c8ede39-a1d6-41f5-9576-3fd866ec551d\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 13:07:01.312095 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:07:01.311935    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5c8ede39-a1d6-41f5-9576-3fd866ec551d-config-out\") pod \"prometheus-k8s-0\" (UID: \"5c8ede39-a1d6-41f5-9576-3fd866ec551d\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 13:07:01.312095 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:07:01.311961    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5c8ede39-a1d6-41f5-9576-3fd866ec551d-web-config\") pod \"prometheus-k8s-0\" (UID: \"5c8ede39-a1d6-41f5-9576-3fd866ec551d\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 13:07:01.312095 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:07:01.312012    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5c8ede39-a1d6-41f5-9576-3fd866ec551d-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5c8ede39-a1d6-41f5-9576-3fd866ec551d\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 13:07:01.312095 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:07:01.312046    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5c8ede39-a1d6-41f5-9576-3fd866ec551d-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"5c8ede39-a1d6-41f5-9576-3fd866ec551d\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 13:07:01.312095 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:07:01.312072    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5c8ede39-a1d6-41f5-9576-3fd866ec551d-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"5c8ede39-a1d6-41f5-9576-3fd866ec551d\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 13:07:01.312324 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:07:01.312100    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/5c8ede39-a1d6-41f5-9576-3fd866ec551d-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"5c8ede39-a1d6-41f5-9576-3fd866ec551d\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 13:07:01.312324 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:07:01.312127    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/5c8ede39-a1d6-41f5-9576-3fd866ec551d-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"5c8ede39-a1d6-41f5-9576-3fd866ec551d\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 13:07:01.312324 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:07:01.312150    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7v94\" (UniqueName: \"kubernetes.io/projected/5c8ede39-a1d6-41f5-9576-3fd866ec551d-kube-api-access-r7v94\") pod \"prometheus-k8s-0\" (UID: \"5c8ede39-a1d6-41f5-9576-3fd866ec551d\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 13:07:01.312324 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:07:01.312175    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/5c8ede39-a1d6-41f5-9576-3fd866ec551d-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"5c8ede39-a1d6-41f5-9576-3fd866ec551d\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 13:07:01.312324 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:07:01.312259    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/5c8ede39-a1d6-41f5-9576-3fd866ec551d-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"5c8ede39-a1d6-41f5-9576-3fd866ec551d\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 13:07:01.312324 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:07:01.312297    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/5c8ede39-a1d6-41f5-9576-3fd866ec551d-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"5c8ede39-a1d6-41f5-9576-3fd866ec551d\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 13:07:01.312594 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:07:01.312386    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/5c8ede39-a1d6-41f5-9576-3fd866ec551d-config\") pod \"prometheus-k8s-0\" (UID: \"5c8ede39-a1d6-41f5-9576-3fd866ec551d\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 13:07:01.312594 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:07:01.312411    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5c8ede39-a1d6-41f5-9576-3fd866ec551d-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5c8ede39-a1d6-41f5-9576-3fd866ec551d\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 13:07:01.312594 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:07:01.312440    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5c8ede39-a1d6-41f5-9576-3fd866ec551d-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5c8ede39-a1d6-41f5-9576-3fd866ec551d\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 13:07:01.413323 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:07:01.413239    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5c8ede39-a1d6-41f5-9576-3fd866ec551d-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"5c8ede39-a1d6-41f5-9576-3fd866ec551d\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 13:07:01.413323 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:07:01.413302    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/5c8ede39-a1d6-41f5-9576-3fd866ec551d-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"5c8ede39-a1d6-41f5-9576-3fd866ec551d\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 13:07:01.413641 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:07:01.413335    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5c8ede39-a1d6-41f5-9576-3fd866ec551d-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"5c8ede39-a1d6-41f5-9576-3fd866ec551d\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 13:07:01.413641 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:07:01.413392    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5c8ede39-a1d6-41f5-9576-3fd866ec551d-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"5c8ede39-a1d6-41f5-9576-3fd866ec551d\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 13:07:01.413641 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:07:01.413427    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5c8ede39-a1d6-41f5-9576-3fd866ec551d-config-out\") pod \"prometheus-k8s-0\" (UID: \"5c8ede39-a1d6-41f5-9576-3fd866ec551d\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 13:07:01.413641 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:07:01.413457    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5c8ede39-a1d6-41f5-9576-3fd866ec551d-web-config\") pod \"prometheus-k8s-0\" (UID: \"5c8ede39-a1d6-41f5-9576-3fd866ec551d\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 13:07:01.413641 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:07:01.413581    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5c8ede39-a1d6-41f5-9576-3fd866ec551d-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5c8ede39-a1d6-41f5-9576-3fd866ec551d\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 13:07:01.413894 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:07:01.413659    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5c8ede39-a1d6-41f5-9576-3fd866ec551d-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"5c8ede39-a1d6-41f5-9576-3fd866ec551d\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 13:07:01.413894 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:07:01.413699    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5c8ede39-a1d6-41f5-9576-3fd866ec551d-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"5c8ede39-a1d6-41f5-9576-3fd866ec551d\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 13:07:01.413894 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:07:01.413728    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/5c8ede39-a1d6-41f5-9576-3fd866ec551d-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"5c8ede39-a1d6-41f5-9576-3fd866ec551d\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 13:07:01.413894 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:07:01.413756    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/5c8ede39-a1d6-41f5-9576-3fd866ec551d-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"5c8ede39-a1d6-41f5-9576-3fd866ec551d\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 13:07:01.413894 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:07:01.413785    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-r7v94\" (UniqueName: \"kubernetes.io/projected/5c8ede39-a1d6-41f5-9576-3fd866ec551d-kube-api-access-r7v94\") pod \"prometheus-k8s-0\" (UID: \"5c8ede39-a1d6-41f5-9576-3fd866ec551d\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 13:07:01.413894 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:07:01.413817    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/5c8ede39-a1d6-41f5-9576-3fd866ec551d-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"5c8ede39-a1d6-41f5-9576-3fd866ec551d\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 13:07:01.413894 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:07:01.413862    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/5c8ede39-a1d6-41f5-9576-3fd866ec551d-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"5c8ede39-a1d6-41f5-9576-3fd866ec551d\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 13:07:01.413894 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:07:01.413889    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/5c8ede39-a1d6-41f5-9576-3fd866ec551d-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"5c8ede39-a1d6-41f5-9576-3fd866ec551d\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 13:07:01.414272 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:07:01.413943    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/5c8ede39-a1d6-41f5-9576-3fd866ec551d-config\") pod \"prometheus-k8s-0\" (UID: \"5c8ede39-a1d6-41f5-9576-3fd866ec551d\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 13:07:01.414272 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:07:01.413967    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5c8ede39-a1d6-41f5-9576-3fd866ec551d-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5c8ede39-a1d6-41f5-9576-3fd866ec551d\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 13:07:01.414272 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:07:01.414003    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5c8ede39-a1d6-41f5-9576-3fd866ec551d-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5c8ede39-a1d6-41f5-9576-3fd866ec551d\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 13:07:01.414272 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:07:01.414205    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5c8ede39-a1d6-41f5-9576-3fd866ec551d-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"5c8ede39-a1d6-41f5-9576-3fd866ec551d\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 13:07:01.414762 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:07:01.414735    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5c8ede39-a1d6-41f5-9576-3fd866ec551d-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5c8ede39-a1d6-41f5-9576-3fd866ec551d\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 13:07:01.417014 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:07:01.416666    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5c8ede39-a1d6-41f5-9576-3fd866ec551d-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5c8ede39-a1d6-41f5-9576-3fd866ec551d\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 13:07:01.417246 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:07:01.417222    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5c8ede39-a1d6-41f5-9576-3fd866ec551d-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"5c8ede39-a1d6-41f5-9576-3fd866ec551d\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 13:07:01.417781 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:07:01.417734    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/5c8ede39-a1d6-41f5-9576-3fd866ec551d-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"5c8ede39-a1d6-41f5-9576-3fd866ec551d\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 13:07:01.418748 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:07:01.418704    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/5c8ede39-a1d6-41f5-9576-3fd866ec551d-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"5c8ede39-a1d6-41f5-9576-3fd866ec551d\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 13:07:01.418898 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:07:01.418876    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/5c8ede39-a1d6-41f5-9576-3fd866ec551d-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"5c8ede39-a1d6-41f5-9576-3fd866ec551d\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 13:07:01.418997 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:07:01.418971    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5c8ede39-a1d6-41f5-9576-3fd866ec551d-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"5c8ede39-a1d6-41f5-9576-3fd866ec551d\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 13:07:01.419254 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:07:01.419232    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/5c8ede39-a1d6-41f5-9576-3fd866ec551d-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"5c8ede39-a1d6-41f5-9576-3fd866ec551d\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 13:07:01.419412 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:07:01.419263    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5c8ede39-a1d6-41f5-9576-3fd866ec551d-config-out\") pod \"prometheus-k8s-0\" (UID: \"5c8ede39-a1d6-41f5-9576-3fd866ec551d\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 13:07:01.419541 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:07:01.419275    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5c8ede39-a1d6-41f5-9576-3fd866ec551d-web-config\") pod \"prometheus-k8s-0\" (UID: \"5c8ede39-a1d6-41f5-9576-3fd866ec551d\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 13:07:01.419668 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:07:01.419627    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5c8ede39-a1d6-41f5-9576-3fd866ec551d-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5c8ede39-a1d6-41f5-9576-3fd866ec551d\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 13:07:01.419668 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:07:01.419310    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/5c8ede39-a1d6-41f5-9576-3fd866ec551d-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"5c8ede39-a1d6-41f5-9576-3fd866ec551d\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 13:07:01.422658 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:07:01.422636    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/5c8ede39-a1d6-41f5-9576-3fd866ec551d-config\") pod \"prometheus-k8s-0\" (UID: \"5c8ede39-a1d6-41f5-9576-3fd866ec551d\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 13:07:01.422767 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:07:01.422719    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/5c8ede39-a1d6-41f5-9576-3fd866ec551d-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"5c8ede39-a1d6-41f5-9576-3fd866ec551d\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 13:07:01.422896 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:07:01.422870    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5c8ede39-a1d6-41f5-9576-3fd866ec551d-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"5c8ede39-a1d6-41f5-9576-3fd866ec551d\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 13:07:01.423715 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:07:01.423693    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5c8ede39-a1d6-41f5-9576-3fd866ec551d-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"5c8ede39-a1d6-41f5-9576-3fd866ec551d\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 13:07:01.426188 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:07:01.426169    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7v94\" (UniqueName: \"kubernetes.io/projected/5c8ede39-a1d6-41f5-9576-3fd866ec551d-kube-api-access-r7v94\") pod \"prometheus-k8s-0\" (UID: \"5c8ede39-a1d6-41f5-9576-3fd866ec551d\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 13:07:01.482846 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:07:01.482808    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
May 25 13:07:01.545750 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:07:01.545706    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-665c9c8584-7s92b" event={"ID":"5db702e4-41d1-42ec-b61d-f71b1cb2fe05","Type":"ContainerStarted","Data":"f03b07f15232ae6a1665db80f396c4713bf4ee85cf44d9ea3e2f0940ec812972"}
May 25 13:07:02.083216 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:07:02.083164    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
May 25 13:07:02.084023 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:07:02.083993    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c8ede39_a1d6_41f5_9576_3fd866ec551d.slice/crio-ac1c45a6b3a9e89332aaccc265d557d8b811e963d0dff46924fe547b62612686 WatchSource:0}: Error finding container ac1c45a6b3a9e89332aaccc265d557d8b811e963d0dff46924fe547b62612686: Status 404 returned error can't find the container with id ac1c45a6b3a9e89332aaccc265d557d8b811e963d0dff46924fe547b62612686
May 25 13:07:02.551838 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:07:02.551795    2557 generic.go:358] "Generic (PLEG): container finished" podID="5c8ede39-a1d6-41f5-9576-3fd866ec551d" containerID="912962ac56818e6f27581a4138b3741d9783befd4fdea3dff2f49bcc83ad5e51" exitCode=0
May 25 13:07:02.552291 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:07:02.551891    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5c8ede39-a1d6-41f5-9576-3fd866ec551d","Type":"ContainerDied","Data":"912962ac56818e6f27581a4138b3741d9783befd4fdea3dff2f49bcc83ad5e51"}
May 25 13:07:02.552291 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:07:02.551939    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5c8ede39-a1d6-41f5-9576-3fd866ec551d","Type":"ContainerStarted","Data":"ac1c45a6b3a9e89332aaccc265d557d8b811e963d0dff46924fe547b62612686"}
May 25 13:07:02.556005 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:07:02.555977    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5f764e8d-9e15-478e-8e6b-50ebf8290b0c","Type":"ContainerStarted","Data":"c502ed26986ae8753b7c8de730bd8a5d89f768d36a51938828df6267fdb07c74"}
May 25 13:07:02.556107 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:07:02.556016    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5f764e8d-9e15-478e-8e6b-50ebf8290b0c","Type":"ContainerStarted","Data":"5aad259ca708fc121eb5b0186d3194369f3bab690061b94be9ed52c70ed6867b"}
May 25 13:07:02.556107 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:07:02.556030    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5f764e8d-9e15-478e-8e6b-50ebf8290b0c","Type":"ContainerStarted","Data":"8599f88f7429773c3df6f8877044d351659770779db4340426126d07030e7ee4"}
May 25 13:07:02.556107 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:07:02.556045    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5f764e8d-9e15-478e-8e6b-50ebf8290b0c","Type":"ContainerStarted","Data":"31d38df8ab3362da1058dfe7c7ef054210aadc334d3b32e5541b942ab2caf53b"}
May 25 13:07:02.556107 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:07:02.556058    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5f764e8d-9e15-478e-8e6b-50ebf8290b0c","Type":"ContainerStarted","Data":"5fc9a14364de7e8b6c6621367f6fbaf1bd5474329b9f150a3e3f2de8a8a253da"}
May 25 13:07:02.556107 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:07:02.556070    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5f764e8d-9e15-478e-8e6b-50ebf8290b0c","Type":"ContainerStarted","Data":"9095031b4e8e17f14bebc8ae5dbcb878bec0e5cd434473ee121374197431d4b7"}
May 25 13:07:02.565206 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:07:02.565158    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-ddc675b94-jt4g7" event={"ID":"1b20cb0e-7386-4292-860f-cae4aa862a94","Type":"ContainerStarted","Data":"159f34ffbfd793bc03f7ffd0c4a4d2d423e023a41503f8d5d687b3486599b893"}
May 25 13:07:02.565206 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:07:02.565210    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-ddc675b94-jt4g7" event={"ID":"1b20cb0e-7386-4292-860f-cae4aa862a94","Type":"ContainerStarted","Data":"fbde3c59852004a7a81cbf8f6beda054da70c50a7f04d0eeaa41316aedf0c65d"}
May 25 13:07:02.565438 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:07:02.565224    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-ddc675b94-jt4g7" event={"ID":"1b20cb0e-7386-4292-860f-cae4aa862a94","Type":"ContainerStarted","Data":"e575c094a35aeabbfb98ab1b5b297b4282abaddd385ed434785bd894833c2d49"}
May 25 13:07:02.565438 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:07:02.565378    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-ddc675b94-jt4g7"
May 25 13:07:02.566647 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:07:02.566618    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-759cd6b97c-kmz7z" event={"ID":"81e544fc-5a66-44c8-8301-0941bedeebe5","Type":"ContainerStarted","Data":"24a972652b5f9fcb25467b8e4819a2a13d041733a252fd711183504c4ed61a4f"}
May 25 13:07:02.602609 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:07:02.602544    2557 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-759cd6b97c-kmz7z" podStartSLOduration=1.8689949019999998 podStartE2EDuration="3.602524756s" podCreationTimestamp="2026-05-25 13:06:59 +0000 UTC" firstStartedPulling="2026-05-25 13:07:00.179114416 +0000 UTC m=+72.642130555" lastFinishedPulling="2026-05-25 13:07:01.912644261 +0000 UTC m=+74.375660409" observedRunningTime="2026-05-25 13:07:02.601338778 +0000 UTC m=+75.064354935" watchObservedRunningTime="2026-05-25 13:07:02.602524756 +0000 UTC m=+75.065540915"
May 25 13:07:02.629543 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:07:02.629484    2557 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.742200143 podStartE2EDuration="7.629465358s" podCreationTimestamp="2026-05-25 13:06:55 +0000 UTC" firstStartedPulling="2026-05-25 13:06:58.023937922 +0000 UTC m=+70.486954060" lastFinishedPulling="2026-05-25 13:07:01.911203129 +0000 UTC m=+74.374219275" observedRunningTime="2026-05-25 13:07:02.62586641 +0000 UTC m=+75.088882568" watchObservedRunningTime="2026-05-25 13:07:02.629465358 +0000 UTC m=+75.092481516"
May 25 13:07:02.650710 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:07:02.650593    2557 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-ddc675b94-jt4g7" podStartSLOduration=2.7912194230000003 podStartE2EDuration="6.65056888s" podCreationTimestamp="2026-05-25 13:06:56 +0000 UTC" firstStartedPulling="2026-05-25 13:06:58.051875357 +0000 UTC m=+70.514891491" lastFinishedPulling="2026-05-25 13:07:01.91122481 +0000 UTC m=+74.374240948" observedRunningTime="2026-05-25 13:07:02.649103712 +0000 UTC m=+75.112119895" watchObservedRunningTime="2026-05-25 13:07:02.65056888 +0000 UTC m=+75.113585038"
May 25 13:07:03.572126 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:07:03.572070    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-665c9c8584-7s92b" event={"ID":"5db702e4-41d1-42ec-b61d-f71b1cb2fe05","Type":"ContainerStarted","Data":"6834d125c5fa02f76f43f921ac534d9fdb69e49143449d917a46a44343d6997f"}
May 25 13:07:03.590406 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:07:03.590323    2557 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-665c9c8584-7s92b" podStartSLOduration=2.98817436 podStartE2EDuration="4.590303417s" podCreationTimestamp="2026-05-25 13:06:59 +0000 UTC" firstStartedPulling="2026-05-25 13:07:01.259370712 +0000 UTC m=+73.722386847" lastFinishedPulling="2026-05-25 13:07:02.861499768 +0000 UTC m=+75.324515904" observedRunningTime="2026-05-25 13:07:03.589808177 +0000 UTC m=+76.052824340" watchObservedRunningTime="2026-05-25 13:07:03.590303417 +0000 UTC m=+76.053319575"
May 25 13:07:04.575086 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:07:04.575045    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-665c9c8584-7s92b"
May 25 13:07:04.580458 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:07:04.580433    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-665c9c8584-7s92b"
May 25 13:07:05.581436 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:07:05.581393    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5c8ede39-a1d6-41f5-9576-3fd866ec551d","Type":"ContainerStarted","Data":"f99e591dc7584d35732273e8d8c7a682731b66b1dcf22f1a288fa68a29fef2a4"}
May 25 13:07:05.581836 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:07:05.581442    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5c8ede39-a1d6-41f5-9576-3fd866ec551d","Type":"ContainerStarted","Data":"a6e6ee876656b10174633d8894f989e05be632d34440af07aaabe619392e144b"}
May 25 13:07:05.581836 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:07:05.581457    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5c8ede39-a1d6-41f5-9576-3fd866ec551d","Type":"ContainerStarted","Data":"a2c383ed9c39a1ad925bf5bc5debfb510ecebef37cb2f0bac34ecc9aa04272ae"}
May 25 13:07:05.581836 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:07:05.581469    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5c8ede39-a1d6-41f5-9576-3fd866ec551d","Type":"ContainerStarted","Data":"28eac03b5efc8b067aa4587d1a3bffc8eac823f6b79217bbf8a3a0a045d26d0b"}
May 25 13:07:05.581836 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:07:05.581479    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5c8ede39-a1d6-41f5-9576-3fd866ec551d","Type":"ContainerStarted","Data":"fbf4f347d4cee0884af1a2dd63e8ac10fe4c0f58337136c0e8a0c8a35f82db1e"}
May 25 13:07:05.581836 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:07:05.581490    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5c8ede39-a1d6-41f5-9576-3fd866ec551d","Type":"ContainerStarted","Data":"09cfc4355960f869bd0d21ae431d52f96c37b4c0d4ea1d595576fa19d89ce87b"}
May 25 13:07:05.614972 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:07:05.614912    2557 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.154265045 podStartE2EDuration="4.614891349s" podCreationTimestamp="2026-05-25 13:07:01 +0000 UTC" firstStartedPulling="2026-05-25 13:07:02.553424247 +0000 UTC m=+75.016440386" lastFinishedPulling="2026-05-25 13:07:05.01405054 +0000 UTC m=+77.477066690" observedRunningTime="2026-05-25 13:07:05.612668657 +0000 UTC m=+78.075684870" watchObservedRunningTime="2026-05-25 13:07:05.614891349 +0000 UTC m=+78.077907507"
May 25 13:07:06.483321 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:07:06.483281    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
May 25 13:07:08.578265 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:07:08.578131    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-ddc675b94-jt4g7"
May 25 13:07:12.385740 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:07:12.385702    2557 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5594bdbb97-l255d"]
May 25 13:07:19.634218 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:07:19.634184    2557 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-759cd6b97c-kmz7z"
May 25 13:07:19.634689 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:07:19.634231    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-759cd6b97c-kmz7z"
May 25 13:07:22.293074 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:07:22.293044    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-5thz9_b8ace320-b067-48f1-9a9d-9f0681b3ee25/serve-healthcheck-canary/0.log"
May 25 13:07:29.534733 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:07:29.534701    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-96mr4"
May 25 13:07:37.411032 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:07:37.410964    2557 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-console/console-5594bdbb97-l255d" podUID="4bb7cd7c-97a8-467d-b7d5-e1db2943618f" containerName="console" containerID="cri-o://8f4ec518d811536ef0b801bcc2902390559fee9158b94b5f682b49862e2487d3" gracePeriod=15
May 25 13:07:37.649852 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:07:37.649825    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-5594bdbb97-l255d_4bb7cd7c-97a8-467d-b7d5-e1db2943618f/console/0.log"
May 25 13:07:37.649991 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:07:37.649905    2557 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5594bdbb97-l255d"
May 25 13:07:37.685032 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:07:37.684956    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-5594bdbb97-l255d_4bb7cd7c-97a8-467d-b7d5-e1db2943618f/console/0.log"
May 25 13:07:37.685032 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:07:37.684999    2557 generic.go:358] "Generic (PLEG): container finished" podID="4bb7cd7c-97a8-467d-b7d5-e1db2943618f" containerID="8f4ec518d811536ef0b801bcc2902390559fee9158b94b5f682b49862e2487d3" exitCode=2
May 25 13:07:37.685252 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:07:37.685032    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5594bdbb97-l255d" event={"ID":"4bb7cd7c-97a8-467d-b7d5-e1db2943618f","Type":"ContainerDied","Data":"8f4ec518d811536ef0b801bcc2902390559fee9158b94b5f682b49862e2487d3"}
May 25 13:07:37.685252 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:07:37.685072    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5594bdbb97-l255d" event={"ID":"4bb7cd7c-97a8-467d-b7d5-e1db2943618f","Type":"ContainerDied","Data":"f39e482e90bff4b099e7e59f1e5780dac0e015c350b6a5a7d15b6a273e78c624"}
May 25 13:07:37.685252 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:07:37.685080    2557 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5594bdbb97-l255d"
May 25 13:07:37.685252 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:07:37.685087    2557 scope.go:117] "RemoveContainer" containerID="8f4ec518d811536ef0b801bcc2902390559fee9158b94b5f682b49862e2487d3"
May 25 13:07:37.694114 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:07:37.694095    2557 scope.go:117] "RemoveContainer" containerID="8f4ec518d811536ef0b801bcc2902390559fee9158b94b5f682b49862e2487d3"
May 25 13:07:37.694438 ip-10-0-143-235 kubenswrapper[2557]: E0525 13:07:37.694416    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f4ec518d811536ef0b801bcc2902390559fee9158b94b5f682b49862e2487d3\": container with ID starting with 8f4ec518d811536ef0b801bcc2902390559fee9158b94b5f682b49862e2487d3 not found: ID does not exist" containerID="8f4ec518d811536ef0b801bcc2902390559fee9158b94b5f682b49862e2487d3"
May 25 13:07:37.694530 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:07:37.694450    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f4ec518d811536ef0b801bcc2902390559fee9158b94b5f682b49862e2487d3"} err="failed to get container status \"8f4ec518d811536ef0b801bcc2902390559fee9158b94b5f682b49862e2487d3\": rpc error: code = NotFound desc = could not find container \"8f4ec518d811536ef0b801bcc2902390559fee9158b94b5f682b49862e2487d3\": container with ID starting with 8f4ec518d811536ef0b801bcc2902390559fee9158b94b5f682b49862e2487d3 not found: ID does not exist"
May 25 13:07:37.737961 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:07:37.737927    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4bb7cd7c-97a8-467d-b7d5-e1db2943618f-oauth-serving-cert\") pod \"4bb7cd7c-97a8-467d-b7d5-e1db2943618f\" (UID: \"4bb7cd7c-97a8-467d-b7d5-e1db2943618f\") "
May 25 13:07:37.737961 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:07:37.737972    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4bb7cd7c-97a8-467d-b7d5-e1db2943618f-trusted-ca-bundle\") pod \"4bb7cd7c-97a8-467d-b7d5-e1db2943618f\" (UID: \"4bb7cd7c-97a8-467d-b7d5-e1db2943618f\") "
May 25 13:07:37.738230 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:07:37.738097    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4bb7cd7c-97a8-467d-b7d5-e1db2943618f-service-ca\") pod \"4bb7cd7c-97a8-467d-b7d5-e1db2943618f\" (UID: \"4bb7cd7c-97a8-467d-b7d5-e1db2943618f\") "
May 25 13:07:37.738230 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:07:37.738134    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4bb7cd7c-97a8-467d-b7d5-e1db2943618f-console-serving-cert\") pod \"4bb7cd7c-97a8-467d-b7d5-e1db2943618f\" (UID: \"4bb7cd7c-97a8-467d-b7d5-e1db2943618f\") "
May 25 13:07:37.738230 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:07:37.738188    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4bb7cd7c-97a8-467d-b7d5-e1db2943618f-console-oauth-config\") pod \"4bb7cd7c-97a8-467d-b7d5-e1db2943618f\" (UID: \"4bb7cd7c-97a8-467d-b7d5-e1db2943618f\") "
May 25 13:07:37.738230 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:07:37.738208    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vxplw\" (UniqueName: \"kubernetes.io/projected/4bb7cd7c-97a8-467d-b7d5-e1db2943618f-kube-api-access-vxplw\") pod \"4bb7cd7c-97a8-467d-b7d5-e1db2943618f\" (UID: \"4bb7cd7c-97a8-467d-b7d5-e1db2943618f\") "
May 25 13:07:37.738477 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:07:37.738234    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4bb7cd7c-97a8-467d-b7d5-e1db2943618f-console-config\") pod \"4bb7cd7c-97a8-467d-b7d5-e1db2943618f\" (UID: \"4bb7cd7c-97a8-467d-b7d5-e1db2943618f\") "
May 25 13:07:37.738477 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:07:37.738324    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4bb7cd7c-97a8-467d-b7d5-e1db2943618f-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "4bb7cd7c-97a8-467d-b7d5-e1db2943618f" (UID: "4bb7cd7c-97a8-467d-b7d5-e1db2943618f"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 13:07:37.738560 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:07:37.738472    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4bb7cd7c-97a8-467d-b7d5-e1db2943618f-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "4bb7cd7c-97a8-467d-b7d5-e1db2943618f" (UID: "4bb7cd7c-97a8-467d-b7d5-e1db2943618f"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 13:07:37.738560 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:07:37.738517    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4bb7cd7c-97a8-467d-b7d5-e1db2943618f-service-ca" (OuterVolumeSpecName: "service-ca") pod "4bb7cd7c-97a8-467d-b7d5-e1db2943618f" (UID: "4bb7cd7c-97a8-467d-b7d5-e1db2943618f"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 13:07:37.738887 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:07:37.738833    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4bb7cd7c-97a8-467d-b7d5-e1db2943618f-console-config" (OuterVolumeSpecName: "console-config") pod "4bb7cd7c-97a8-467d-b7d5-e1db2943618f" (UID: "4bb7cd7c-97a8-467d-b7d5-e1db2943618f"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 13:07:37.740497 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:07:37.740467    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4bb7cd7c-97a8-467d-b7d5-e1db2943618f-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "4bb7cd7c-97a8-467d-b7d5-e1db2943618f" (UID: "4bb7cd7c-97a8-467d-b7d5-e1db2943618f"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 13:07:37.740571 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:07:37.740489    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4bb7cd7c-97a8-467d-b7d5-e1db2943618f-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "4bb7cd7c-97a8-467d-b7d5-e1db2943618f" (UID: "4bb7cd7c-97a8-467d-b7d5-e1db2943618f"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 13:07:37.740571 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:07:37.740499    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4bb7cd7c-97a8-467d-b7d5-e1db2943618f-kube-api-access-vxplw" (OuterVolumeSpecName: "kube-api-access-vxplw") pod "4bb7cd7c-97a8-467d-b7d5-e1db2943618f" (UID: "4bb7cd7c-97a8-467d-b7d5-e1db2943618f"). InnerVolumeSpecName "kube-api-access-vxplw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 13:07:37.839141 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:07:37.839103    2557 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4bb7cd7c-97a8-467d-b7d5-e1db2943618f-service-ca\") on node \"ip-10-0-143-235.ec2.internal\" DevicePath \"\""
May 25 13:07:37.839141 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:07:37.839132    2557 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4bb7cd7c-97a8-467d-b7d5-e1db2943618f-console-serving-cert\") on node \"ip-10-0-143-235.ec2.internal\" DevicePath \"\""
May 25 13:07:37.839141 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:07:37.839144    2557 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4bb7cd7c-97a8-467d-b7d5-e1db2943618f-console-oauth-config\") on node \"ip-10-0-143-235.ec2.internal\" DevicePath \"\""
May 25 13:07:37.839386 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:07:37.839153    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vxplw\" (UniqueName: \"kubernetes.io/projected/4bb7cd7c-97a8-467d-b7d5-e1db2943618f-kube-api-access-vxplw\") on node \"ip-10-0-143-235.ec2.internal\" DevicePath \"\""
May 25 13:07:37.839386 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:07:37.839164    2557 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4bb7cd7c-97a8-467d-b7d5-e1db2943618f-console-config\") on node \"ip-10-0-143-235.ec2.internal\" DevicePath \"\""
May 25 13:07:37.839386 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:07:37.839173    2557 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4bb7cd7c-97a8-467d-b7d5-e1db2943618f-oauth-serving-cert\") on node \"ip-10-0-143-235.ec2.internal\" DevicePath \"\""
May 25 13:07:37.839386 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:07:37.839181    2557 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4bb7cd7c-97a8-467d-b7d5-e1db2943618f-trusted-ca-bundle\") on node \"ip-10-0-143-235.ec2.internal\" DevicePath \"\""
May 25 13:07:38.010073 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:07:38.010027    2557 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5594bdbb97-l255d"]
May 25 13:07:38.013249 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:07:38.013215    2557 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5594bdbb97-l255d"]
May 25 13:07:38.149717 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:07:38.149687    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4bb7cd7c-97a8-467d-b7d5-e1db2943618f" path="/var/lib/kubelet/pods/4bb7cd7c-97a8-467d-b7d5-e1db2943618f/volumes"
May 25 13:07:39.639652 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:07:39.639621    2557 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-759cd6b97c-kmz7z"
May 25 13:07:39.643594 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:07:39.643566    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-759cd6b97c-kmz7z"
May 25 13:08:01.483209 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:08:01.483155    2557 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
May 25 13:08:01.503374 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:08:01.503324    2557 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
May 25 13:08:01.776252 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:08:01.776177    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
May 25 13:08:19.210370 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:08:19.210318    2557 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-6bb4767667-qjqdw"]
May 25 13:08:19.210961 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:08:19.210793    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4bb7cd7c-97a8-467d-b7d5-e1db2943618f" containerName="console"
May 25 13:08:19.210961 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:08:19.210809    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="4bb7cd7c-97a8-467d-b7d5-e1db2943618f" containerName="console"
May 25 13:08:19.210961 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:08:19.210890    2557 memory_manager.go:356] "RemoveStaleState removing state" podUID="4bb7cd7c-97a8-467d-b7d5-e1db2943618f" containerName="console"
May 25 13:08:19.213730 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:08:19.213709    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-6bb4767667-qjqdw"
May 25 13:08:19.216381 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:08:19.216341    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
May 25 13:08:19.217034 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:08:19.216524    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
May 25 13:08:19.217298 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:08:19.216689    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-v6xzp\""
May 25 13:08:19.217580 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:08:19.216762    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
May 25 13:08:19.218668 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:08:19.216802    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
May 25 13:08:19.218770 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:08:19.216845    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
May 25 13:08:19.222397 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:08:19.222365    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
May 25 13:08:19.231617 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:08:19.231584    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-6bb4767667-qjqdw"]
May 25 13:08:19.288208 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:08:19.288168    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/1e74c703-bb22-4ac6-8e96-1b429b819371-telemeter-client-tls\") pod \"telemeter-client-6bb4767667-qjqdw\" (UID: \"1e74c703-bb22-4ac6-8e96-1b429b819371\") " pod="openshift-monitoring/telemeter-client-6bb4767667-qjqdw"
May 25 13:08:19.288428 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:08:19.288219    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9hps\" (UniqueName: \"kubernetes.io/projected/1e74c703-bb22-4ac6-8e96-1b429b819371-kube-api-access-t9hps\") pod \"telemeter-client-6bb4767667-qjqdw\" (UID: \"1e74c703-bb22-4ac6-8e96-1b429b819371\") " pod="openshift-monitoring/telemeter-client-6bb4767667-qjqdw"
May 25 13:08:19.288428 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:08:19.288248    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/1e74c703-bb22-4ac6-8e96-1b429b819371-secret-telemeter-client\") pod \"telemeter-client-6bb4767667-qjqdw\" (UID: \"1e74c703-bb22-4ac6-8e96-1b429b819371\") " pod="openshift-monitoring/telemeter-client-6bb4767667-qjqdw"
May 25 13:08:19.288428 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:08:19.288266    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/1e74c703-bb22-4ac6-8e96-1b429b819371-federate-client-tls\") pod \"telemeter-client-6bb4767667-qjqdw\" (UID: \"1e74c703-bb22-4ac6-8e96-1b429b819371\") " pod="openshift-monitoring/telemeter-client-6bb4767667-qjqdw"
May 25 13:08:19.288428 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:08:19.288285    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1e74c703-bb22-4ac6-8e96-1b429b819371-serving-certs-ca-bundle\") pod \"telemeter-client-6bb4767667-qjqdw\" (UID: \"1e74c703-bb22-4ac6-8e96-1b429b819371\") " pod="openshift-monitoring/telemeter-client-6bb4767667-qjqdw"
May 25 13:08:19.288428 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:08:19.288307    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1e74c703-bb22-4ac6-8e96-1b429b819371-metrics-client-ca\") pod \"telemeter-client-6bb4767667-qjqdw\" (UID: \"1e74c703-bb22-4ac6-8e96-1b429b819371\") " pod="openshift-monitoring/telemeter-client-6bb4767667-qjqdw"
May 25 13:08:19.288428 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:08:19.288325    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1e74c703-bb22-4ac6-8e96-1b429b819371-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6bb4767667-qjqdw\" (UID: \"1e74c703-bb22-4ac6-8e96-1b429b819371\") " pod="openshift-monitoring/telemeter-client-6bb4767667-qjqdw"
May 25 13:08:19.288428 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:08:19.288389    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1e74c703-bb22-4ac6-8e96-1b429b819371-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6bb4767667-qjqdw\" (UID: \"1e74c703-bb22-4ac6-8e96-1b429b819371\") " pod="openshift-monitoring/telemeter-client-6bb4767667-qjqdw"
May 25 13:08:19.389541 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:08:19.389498    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1e74c703-bb22-4ac6-8e96-1b429b819371-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6bb4767667-qjqdw\" (UID: \"1e74c703-bb22-4ac6-8e96-1b429b819371\") " pod="openshift-monitoring/telemeter-client-6bb4767667-qjqdw"
May 25 13:08:19.389733 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:08:19.389569    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/1e74c703-bb22-4ac6-8e96-1b429b819371-telemeter-client-tls\") pod \"telemeter-client-6bb4767667-qjqdw\" (UID: \"1e74c703-bb22-4ac6-8e96-1b429b819371\") " pod="openshift-monitoring/telemeter-client-6bb4767667-qjqdw"
May 25 13:08:19.389733 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:08:19.389604    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-t9hps\" (UniqueName: \"kubernetes.io/projected/1e74c703-bb22-4ac6-8e96-1b429b819371-kube-api-access-t9hps\") pod \"telemeter-client-6bb4767667-qjqdw\" (UID: \"1e74c703-bb22-4ac6-8e96-1b429b819371\") " pod="openshift-monitoring/telemeter-client-6bb4767667-qjqdw"
May 25 13:08:19.389733 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:08:19.389631    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/1e74c703-bb22-4ac6-8e96-1b429b819371-secret-telemeter-client\") pod \"telemeter-client-6bb4767667-qjqdw\" (UID: \"1e74c703-bb22-4ac6-8e96-1b429b819371\") " pod="openshift-monitoring/telemeter-client-6bb4767667-qjqdw"
May 25 13:08:19.389733 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:08:19.389649    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/1e74c703-bb22-4ac6-8e96-1b429b819371-federate-client-tls\") pod \"telemeter-client-6bb4767667-qjqdw\" (UID: \"1e74c703-bb22-4ac6-8e96-1b429b819371\") " pod="openshift-monitoring/telemeter-client-6bb4767667-qjqdw"
May 25 13:08:19.389733 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:08:19.389665    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1e74c703-bb22-4ac6-8e96-1b429b819371-serving-certs-ca-bundle\") pod \"telemeter-client-6bb4767667-qjqdw\" (UID: \"1e74c703-bb22-4ac6-8e96-1b429b819371\") " pod="openshift-monitoring/telemeter-client-6bb4767667-qjqdw"
May 25 13:08:19.389733 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:08:19.389687    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1e74c703-bb22-4ac6-8e96-1b429b819371-metrics-client-ca\") pod \"telemeter-client-6bb4767667-qjqdw\" (UID: \"1e74c703-bb22-4ac6-8e96-1b429b819371\") " pod="openshift-monitoring/telemeter-client-6bb4767667-qjqdw"
May 25 13:08:19.390019 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:08:19.389734    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1e74c703-bb22-4ac6-8e96-1b429b819371-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6bb4767667-qjqdw\" (UID: \"1e74c703-bb22-4ac6-8e96-1b429b819371\") " pod="openshift-monitoring/telemeter-client-6bb4767667-qjqdw"
May 25 13:08:19.390405 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:08:19.390384    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1e74c703-bb22-4ac6-8e96-1b429b819371-serving-certs-ca-bundle\") pod \"telemeter-client-6bb4767667-qjqdw\" (UID: \"1e74c703-bb22-4ac6-8e96-1b429b819371\") " pod="openshift-monitoring/telemeter-client-6bb4767667-qjqdw"
May 25 13:08:19.390545 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:08:19.390503    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1e74c703-bb22-4ac6-8e96-1b429b819371-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6bb4767667-qjqdw\" (UID: \"1e74c703-bb22-4ac6-8e96-1b429b819371\") " pod="openshift-monitoring/telemeter-client-6bb4767667-qjqdw"
May 25 13:08:19.390628 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:08:19.390608    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1e74c703-bb22-4ac6-8e96-1b429b819371-metrics-client-ca\") pod \"telemeter-client-6bb4767667-qjqdw\" (UID: \"1e74c703-bb22-4ac6-8e96-1b429b819371\") " pod="openshift-monitoring/telemeter-client-6bb4767667-qjqdw"
May 25 13:08:19.392269 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:08:19.392243    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1e74c703-bb22-4ac6-8e96-1b429b819371-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6bb4767667-qjqdw\" (UID: \"1e74c703-bb22-4ac6-8e96-1b429b819371\") " pod="openshift-monitoring/telemeter-client-6bb4767667-qjqdw"
May 25 13:08:19.392389 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:08:19.392332    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/1e74c703-bb22-4ac6-8e96-1b429b819371-secret-telemeter-client\") pod \"telemeter-client-6bb4767667-qjqdw\" (UID: \"1e74c703-bb22-4ac6-8e96-1b429b819371\") " pod="openshift-monitoring/telemeter-client-6bb4767667-qjqdw"
May 25 13:08:19.392578 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:08:19.392561    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/1e74c703-bb22-4ac6-8e96-1b429b819371-telemeter-client-tls\") pod \"telemeter-client-6bb4767667-qjqdw\" (UID: \"1e74c703-bb22-4ac6-8e96-1b429b819371\") " pod="openshift-monitoring/telemeter-client-6bb4767667-qjqdw"
May 25 13:08:19.392638 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:08:19.392564    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/1e74c703-bb22-4ac6-8e96-1b429b819371-federate-client-tls\") pod \"telemeter-client-6bb4767667-qjqdw\" (UID: \"1e74c703-bb22-4ac6-8e96-1b429b819371\") " pod="openshift-monitoring/telemeter-client-6bb4767667-qjqdw"
May 25 13:08:19.397727 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:08:19.397695    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9hps\" (UniqueName: \"kubernetes.io/projected/1e74c703-bb22-4ac6-8e96-1b429b819371-kube-api-access-t9hps\") pod \"telemeter-client-6bb4767667-qjqdw\" (UID: \"1e74c703-bb22-4ac6-8e96-1b429b819371\") " pod="openshift-monitoring/telemeter-client-6bb4767667-qjqdw"
May 25 13:08:19.525103 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:08:19.525000    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-6bb4767667-qjqdw"
May 25 13:08:19.659965 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:08:19.659936    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-6bb4767667-qjqdw"]
May 25 13:08:19.662306 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:08:19.662276    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e74c703_bb22_4ac6_8e96_1b429b819371.slice/crio-278d12b8999bd77d11b720152aad62b5a43c483a150f9c2e0b3b63cf29abd46c WatchSource:0}: Error finding container 278d12b8999bd77d11b720152aad62b5a43c483a150f9c2e0b3b63cf29abd46c: Status 404 returned error can't find the container with id 278d12b8999bd77d11b720152aad62b5a43c483a150f9c2e0b3b63cf29abd46c
May 25 13:08:19.811531 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:08:19.811490    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6bb4767667-qjqdw" event={"ID":"1e74c703-bb22-4ac6-8e96-1b429b819371","Type":"ContainerStarted","Data":"278d12b8999bd77d11b720152aad62b5a43c483a150f9c2e0b3b63cf29abd46c"}
May 25 13:08:21.820651 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:08:21.820617    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6bb4767667-qjqdw" event={"ID":"1e74c703-bb22-4ac6-8e96-1b429b819371","Type":"ContainerStarted","Data":"47ae1a8d9c52fcb66aa44324f29b539ba26d849058022a792f2a37be656f436c"}
May 25 13:08:21.820651 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:08:21.820652    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6bb4767667-qjqdw" event={"ID":"1e74c703-bb22-4ac6-8e96-1b429b819371","Type":"ContainerStarted","Data":"f13079c4c503fc452931f27fd5e5389e427b61de5b9abc6f49e1bb8f31b6756d"}
May 25 13:08:21.821084 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:08:21.820666    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6bb4767667-qjqdw" event={"ID":"1e74c703-bb22-4ac6-8e96-1b429b819371","Type":"ContainerStarted","Data":"0a52d0d9dd471de391d2ea0142a2981d9acb4ec446ebbfeb5dc779ecdd47f2d7"}
May 25 13:08:21.844180 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:08:21.844117    2557 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-6bb4767667-qjqdw" podStartSLOduration=1.218240515 podStartE2EDuration="2.844097416s" podCreationTimestamp="2026-05-25 13:08:19 +0000 UTC" firstStartedPulling="2026-05-25 13:08:19.664176584 +0000 UTC m=+152.127192719" lastFinishedPulling="2026-05-25 13:08:21.290033481 +0000 UTC m=+153.753049620" observedRunningTime="2026-05-25 13:08:21.84224449 +0000 UTC m=+154.305260649" watchObservedRunningTime="2026-05-25 13:08:21.844097416 +0000 UTC m=+154.307113585"
May 25 13:08:57.295525 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:08:57.295439    2557 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-88vxt"]
May 25 13:08:57.298853 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:08:57.298832    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-88vxt"
May 25 13:08:57.301485 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:08:57.301464    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
May 25 13:08:57.305757 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:08:57.305731    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-88vxt"]
May 25 13:08:57.321933 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:08:57.321895    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/81b07008-64d7-4948-a23d-4eaae2c55da5-dbus\") pod \"global-pull-secret-syncer-88vxt\" (UID: \"81b07008-64d7-4948-a23d-4eaae2c55da5\") " pod="kube-system/global-pull-secret-syncer-88vxt"
May 25 13:08:57.322179 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:08:57.322159    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/81b07008-64d7-4948-a23d-4eaae2c55da5-kubelet-config\") pod \"global-pull-secret-syncer-88vxt\" (UID: \"81b07008-64d7-4948-a23d-4eaae2c55da5\") " pod="kube-system/global-pull-secret-syncer-88vxt"
May 25 13:08:57.322310 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:08:57.322295    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/81b07008-64d7-4948-a23d-4eaae2c55da5-original-pull-secret\") pod \"global-pull-secret-syncer-88vxt\" (UID: \"81b07008-64d7-4948-a23d-4eaae2c55da5\") " pod="kube-system/global-pull-secret-syncer-88vxt"
May 25 13:08:57.423894 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:08:57.423846    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/81b07008-64d7-4948-a23d-4eaae2c55da5-dbus\") pod \"global-pull-secret-syncer-88vxt\" (UID: \"81b07008-64d7-4948-a23d-4eaae2c55da5\") " pod="kube-system/global-pull-secret-syncer-88vxt"
May 25 13:08:57.424061 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:08:57.423902    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/81b07008-64d7-4948-a23d-4eaae2c55da5-kubelet-config\") pod \"global-pull-secret-syncer-88vxt\" (UID: \"81b07008-64d7-4948-a23d-4eaae2c55da5\") " pod="kube-system/global-pull-secret-syncer-88vxt"
May 25 13:08:57.424061 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:08:57.423936    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/81b07008-64d7-4948-a23d-4eaae2c55da5-original-pull-secret\") pod \"global-pull-secret-syncer-88vxt\" (UID: \"81b07008-64d7-4948-a23d-4eaae2c55da5\") " pod="kube-system/global-pull-secret-syncer-88vxt"
May 25 13:08:57.424129 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:08:57.424054    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/81b07008-64d7-4948-a23d-4eaae2c55da5-kubelet-config\") pod \"global-pull-secret-syncer-88vxt\" (UID: \"81b07008-64d7-4948-a23d-4eaae2c55da5\") " pod="kube-system/global-pull-secret-syncer-88vxt"
May 25 13:08:57.424129 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:08:57.424066    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/81b07008-64d7-4948-a23d-4eaae2c55da5-dbus\") pod \"global-pull-secret-syncer-88vxt\" (UID: \"81b07008-64d7-4948-a23d-4eaae2c55da5\") " pod="kube-system/global-pull-secret-syncer-88vxt"
May 25 13:08:57.426381 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:08:57.426336    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/81b07008-64d7-4948-a23d-4eaae2c55da5-original-pull-secret\") pod \"global-pull-secret-syncer-88vxt\" (UID: \"81b07008-64d7-4948-a23d-4eaae2c55da5\") " pod="kube-system/global-pull-secret-syncer-88vxt"
May 25 13:08:57.608757 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:08:57.608722    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-88vxt"
May 25 13:08:57.737525 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:08:57.737499    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-88vxt"]
May 25 13:08:57.739932 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:08:57.739903    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81b07008_64d7_4948_a23d_4eaae2c55da5.slice/crio-ed4d64d6c834290cef1b6550aac01a848b3a41fa3b2d48774b7f18886f74f190 WatchSource:0}: Error finding container ed4d64d6c834290cef1b6550aac01a848b3a41fa3b2d48774b7f18886f74f190: Status 404 returned error can't find the container with id ed4d64d6c834290cef1b6550aac01a848b3a41fa3b2d48774b7f18886f74f190
May 25 13:08:57.932264 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:08:57.932172    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-88vxt" event={"ID":"81b07008-64d7-4948-a23d-4eaae2c55da5","Type":"ContainerStarted","Data":"ed4d64d6c834290cef1b6550aac01a848b3a41fa3b2d48774b7f18886f74f190"}
May 25 13:09:01.946163 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:09:01.946091    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-88vxt" event={"ID":"81b07008-64d7-4948-a23d-4eaae2c55da5","Type":"ContainerStarted","Data":"5bdb7feec818f557cfdf480cb730d6467d5ca62f38ba4c7e05acf1a9946d926b"}
May 25 13:09:01.961508 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:09:01.961435    2557 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-88vxt" podStartSLOduration=1.193282358 podStartE2EDuration="4.961416819s" podCreationTimestamp="2026-05-25 13:08:57 +0000 UTC" firstStartedPulling="2026-05-25 13:08:57.741575197 +0000 UTC m=+190.204591332" lastFinishedPulling="2026-05-25 13:09:01.509709654 +0000 UTC m=+193.972725793" observedRunningTime="2026-05-25 13:09:01.961133249 +0000 UTC m=+194.424149407" watchObservedRunningTime="2026-05-25 13:09:01.961416819 +0000 UTC m=+194.424432977"
May 25 13:10:48.035534 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:10:48.035502    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vqnvc_d21bdf8b-d0ab-48f2-8736-54e92947dd53/ovn-acl-logging/0.log"
May 25 13:10:48.036080 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:10:48.035821    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vqnvc_d21bdf8b-d0ab-48f2-8736-54e92947dd53/ovn-acl-logging/0.log"
May 25 13:10:48.044523 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:10:48.044497    2557 kubelet.go:1628] "Image garbage collection succeeded"
May 25 13:15:48.061928 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:15:48.061896    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vqnvc_d21bdf8b-d0ab-48f2-8736-54e92947dd53/ovn-acl-logging/0.log"
May 25 13:15:48.062964 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:15:48.062943    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vqnvc_d21bdf8b-d0ab-48f2-8736-54e92947dd53/ovn-acl-logging/0.log"
May 25 13:20:10.829068 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:20:10.829034    2557 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-587ccfb98-f5kjv"]
May 25 13:20:10.831694 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:20:10.831677    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-587ccfb98-f5kjv"
May 25 13:20:10.834519 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:20:10.834497    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
May 25 13:20:10.835638 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:20:10.835602    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-6swzc\""
May 25 13:20:10.835638 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:20:10.835605    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
May 25 13:20:10.852459 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:20:10.852429    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-587ccfb98-f5kjv"]
May 25 13:20:10.916603 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:20:10.916565    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmtt7\" (UniqueName: \"kubernetes.io/projected/9472c8b8-c748-4cd5-ac4a-914824439d8e-kube-api-access-mmtt7\") pod \"cert-manager-webhook-587ccfb98-f5kjv\" (UID: \"9472c8b8-c748-4cd5-ac4a-914824439d8e\") " pod="cert-manager/cert-manager-webhook-587ccfb98-f5kjv"
May 25 13:20:10.916777 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:20:10.916669    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9472c8b8-c748-4cd5-ac4a-914824439d8e-bound-sa-token\") pod \"cert-manager-webhook-587ccfb98-f5kjv\" (UID: \"9472c8b8-c748-4cd5-ac4a-914824439d8e\") " pod="cert-manager/cert-manager-webhook-587ccfb98-f5kjv"
May 25 13:20:11.017994 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:20:11.017953    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9472c8b8-c748-4cd5-ac4a-914824439d8e-bound-sa-token\") pod \"cert-manager-webhook-587ccfb98-f5kjv\" (UID: \"9472c8b8-c748-4cd5-ac4a-914824439d8e\") " pod="cert-manager/cert-manager-webhook-587ccfb98-f5kjv"
May 25 13:20:11.018173 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:20:11.018030    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-mmtt7\" (UniqueName: \"kubernetes.io/projected/9472c8b8-c748-4cd5-ac4a-914824439d8e-kube-api-access-mmtt7\") pod \"cert-manager-webhook-587ccfb98-f5kjv\" (UID: \"9472c8b8-c748-4cd5-ac4a-914824439d8e\") " pod="cert-manager/cert-manager-webhook-587ccfb98-f5kjv"
May 25 13:20:11.027444 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:20:11.027410    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9472c8b8-c748-4cd5-ac4a-914824439d8e-bound-sa-token\") pod \"cert-manager-webhook-587ccfb98-f5kjv\" (UID: \"9472c8b8-c748-4cd5-ac4a-914824439d8e\") " pod="cert-manager/cert-manager-webhook-587ccfb98-f5kjv"
May 25 13:20:11.027576 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:20:11.027564    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmtt7\" (UniqueName: \"kubernetes.io/projected/9472c8b8-c748-4cd5-ac4a-914824439d8e-kube-api-access-mmtt7\") pod \"cert-manager-webhook-587ccfb98-f5kjv\" (UID: \"9472c8b8-c748-4cd5-ac4a-914824439d8e\") " pod="cert-manager/cert-manager-webhook-587ccfb98-f5kjv"
May 25 13:20:11.152058 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:20:11.151970    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-587ccfb98-f5kjv"
May 25 13:20:11.285199 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:20:11.285095    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-587ccfb98-f5kjv"]
May 25 13:20:11.288155 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:20:11.288126    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9472c8b8_c748_4cd5_ac4a_914824439d8e.slice/crio-6b0e6e409d6b44d1180d2e5c77975107de541a8053626f812e7cc50585ec657d WatchSource:0}: Error finding container 6b0e6e409d6b44d1180d2e5c77975107de541a8053626f812e7cc50585ec657d: Status 404 returned error can't find the container with id 6b0e6e409d6b44d1180d2e5c77975107de541a8053626f812e7cc50585ec657d
May 25 13:20:11.289932 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:20:11.289914    2557 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 25 13:20:11.654484 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:20:11.654454    2557 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-68b757865b-w4z5b"]
May 25 13:20:11.656968 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:20:11.656945    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-68b757865b-w4z5b"
May 25 13:20:11.659638 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:20:11.659619    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-g8vk4\""
May 25 13:20:11.667534 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:20:11.667511    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-68b757865b-w4z5b"]
May 25 13:20:11.724143 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:20:11.724106    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lw5mk\" (UniqueName: \"kubernetes.io/projected/b3bc7587-f757-4029-b622-1ff2e0e61e47-kube-api-access-lw5mk\") pod \"cert-manager-cainjector-68b757865b-w4z5b\" (UID: \"b3bc7587-f757-4029-b622-1ff2e0e61e47\") " pod="cert-manager/cert-manager-cainjector-68b757865b-w4z5b"
May 25 13:20:11.724331 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:20:11.724248    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b3bc7587-f757-4029-b622-1ff2e0e61e47-bound-sa-token\") pod \"cert-manager-cainjector-68b757865b-w4z5b\" (UID: \"b3bc7587-f757-4029-b622-1ff2e0e61e47\") " pod="cert-manager/cert-manager-cainjector-68b757865b-w4z5b"
May 25 13:20:11.824820 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:20:11.824778    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b3bc7587-f757-4029-b622-1ff2e0e61e47-bound-sa-token\") pod \"cert-manager-cainjector-68b757865b-w4z5b\" (UID: \"b3bc7587-f757-4029-b622-1ff2e0e61e47\") " pod="cert-manager/cert-manager-cainjector-68b757865b-w4z5b"
May 25 13:20:11.825008 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:20:11.824834    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-lw5mk\" (UniqueName: \"kubernetes.io/projected/b3bc7587-f757-4029-b622-1ff2e0e61e47-kube-api-access-lw5mk\") pod \"cert-manager-cainjector-68b757865b-w4z5b\" (UID: \"b3bc7587-f757-4029-b622-1ff2e0e61e47\") " pod="cert-manager/cert-manager-cainjector-68b757865b-w4z5b"
May 25 13:20:11.835175 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:20:11.835149    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b3bc7587-f757-4029-b622-1ff2e0e61e47-bound-sa-token\") pod \"cert-manager-cainjector-68b757865b-w4z5b\" (UID: \"b3bc7587-f757-4029-b622-1ff2e0e61e47\") " pod="cert-manager/cert-manager-cainjector-68b757865b-w4z5b"
May 25 13:20:11.835873 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:20:11.835853    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lw5mk\" (UniqueName: \"kubernetes.io/projected/b3bc7587-f757-4029-b622-1ff2e0e61e47-kube-api-access-lw5mk\") pod \"cert-manager-cainjector-68b757865b-w4z5b\" (UID: \"b3bc7587-f757-4029-b622-1ff2e0e61e47\") " pod="cert-manager/cert-manager-cainjector-68b757865b-w4z5b"
May 25 13:20:11.961909 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:20:11.961816    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-587ccfb98-f5kjv" event={"ID":"9472c8b8-c748-4cd5-ac4a-914824439d8e","Type":"ContainerStarted","Data":"6b0e6e409d6b44d1180d2e5c77975107de541a8053626f812e7cc50585ec657d"}
May 25 13:20:11.967117 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:20:11.967098    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-68b757865b-w4z5b"
May 25 13:20:12.107908 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:20:12.107849    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-68b757865b-w4z5b"]
May 25 13:20:12.112099 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:20:12.112065    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb3bc7587_f757_4029_b622_1ff2e0e61e47.slice/crio-f7d023eba62be3db2e6eca602484a220fcfd3ad7f94c873fffd7c12bfa7a32ac WatchSource:0}: Error finding container f7d023eba62be3db2e6eca602484a220fcfd3ad7f94c873fffd7c12bfa7a32ac: Status 404 returned error can't find the container with id f7d023eba62be3db2e6eca602484a220fcfd3ad7f94c873fffd7c12bfa7a32ac
May 25 13:20:12.967516 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:20:12.967462    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-68b757865b-w4z5b" event={"ID":"b3bc7587-f757-4029-b622-1ff2e0e61e47","Type":"ContainerStarted","Data":"f7d023eba62be3db2e6eca602484a220fcfd3ad7f94c873fffd7c12bfa7a32ac"}
May 25 13:20:14.975196 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:20:14.975092    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-68b757865b-w4z5b" event={"ID":"b3bc7587-f757-4029-b622-1ff2e0e61e47","Type":"ContainerStarted","Data":"42941fea00e733ab0ec1345497c0d128433067639d32915105f07521a9b189a4"}
May 25 13:20:14.976404 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:20:14.976378    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-587ccfb98-f5kjv" event={"ID":"9472c8b8-c748-4cd5-ac4a-914824439d8e","Type":"ContainerStarted","Data":"4806ca8cb465da2b9e7d888d9d7f43b764363cd3eb15f45367139b48855a71f7"}
May 25 13:20:14.976507 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:20:14.976472    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-587ccfb98-f5kjv"
May 25 13:20:14.992779 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:20:14.992728    2557 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-68b757865b-w4z5b" podStartSLOduration=1.384813614 podStartE2EDuration="3.992711193s" podCreationTimestamp="2026-05-25 13:20:11 +0000 UTC" firstStartedPulling="2026-05-25 13:20:12.114787177 +0000 UTC m=+864.577803328" lastFinishedPulling="2026-05-25 13:20:14.722684769 +0000 UTC m=+867.185700907" observedRunningTime="2026-05-25 13:20:14.99132408 +0000 UTC m=+867.454340237" watchObservedRunningTime="2026-05-25 13:20:14.992711193 +0000 UTC m=+867.455727349"
May 25 13:20:15.014938 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:20:15.014723    2557 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-587ccfb98-f5kjv" podStartSLOduration=1.591413338 podStartE2EDuration="5.014709259s" podCreationTimestamp="2026-05-25 13:20:10 +0000 UTC" firstStartedPulling="2026-05-25 13:20:11.290038589 +0000 UTC m=+863.753054724" lastFinishedPulling="2026-05-25 13:20:14.713334509 +0000 UTC m=+867.176350645" observedRunningTime="2026-05-25 13:20:15.014034047 +0000 UTC m=+867.477050220" watchObservedRunningTime="2026-05-25 13:20:15.014709259 +0000 UTC m=+867.477725416"
May 25 13:20:20.982431 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:20:20.982401    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-587ccfb98-f5kjv"
May 25 13:20:48.084571 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:20:48.084490    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vqnvc_d21bdf8b-d0ab-48f2-8736-54e92947dd53/ovn-acl-logging/0.log"
May 25 13:20:48.086412 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:20:48.086390    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vqnvc_d21bdf8b-d0ab-48f2-8736-54e92947dd53/ovn-acl-logging/0.log"
May 25 13:20:56.864424 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:20:56.864386    2557 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-6f6cfd7d7-s6cd2"]
May 25 13:20:56.867254 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:20:56.867223    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6f6cfd7d7-s6cd2"
May 25 13:20:56.873074 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:20:56.873046    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
May 25 13:20:56.873255 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:20:56.873050    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-dbmdq\""
May 25 13:20:56.873393 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:20:56.873377    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
May 25 13:20:56.873472 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:20:56.873447    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
May 25 13:20:56.873528 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:20:56.873495    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
May 25 13:20:56.874027 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:20:56.874009    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
May 25 13:20:56.885582 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:20:56.885548    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6f6cfd7d7-s6cd2"]
May 25 13:20:57.028942 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:20:57.028907    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b920cc63-c662-4048-a720-68be65daa919-cert\") pod \"lws-controller-manager-6f6cfd7d7-s6cd2\" (UID: \"b920cc63-c662-4048-a720-68be65daa919\") " pod="openshift-lws-operator/lws-controller-manager-6f6cfd7d7-s6cd2"
May 25 13:20:57.028942 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:20:57.028947    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8d6zz\" (UniqueName: \"kubernetes.io/projected/b920cc63-c662-4048-a720-68be65daa919-kube-api-access-8d6zz\") pod \"lws-controller-manager-6f6cfd7d7-s6cd2\" (UID: \"b920cc63-c662-4048-a720-68be65daa919\") " pod="openshift-lws-operator/lws-controller-manager-6f6cfd7d7-s6cd2"
May 25 13:20:57.029215 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:20:57.028992    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/b920cc63-c662-4048-a720-68be65daa919-manager-config\") pod \"lws-controller-manager-6f6cfd7d7-s6cd2\" (UID: \"b920cc63-c662-4048-a720-68be65daa919\") " pod="openshift-lws-operator/lws-controller-manager-6f6cfd7d7-s6cd2"
May 25 13:20:57.029215 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:20:57.029075    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/b920cc63-c662-4048-a720-68be65daa919-metrics-cert\") pod \"lws-controller-manager-6f6cfd7d7-s6cd2\" (UID: \"b920cc63-c662-4048-a720-68be65daa919\") " pod="openshift-lws-operator/lws-controller-manager-6f6cfd7d7-s6cd2"
May 25 13:20:57.129794 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:20:57.129705    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b920cc63-c662-4048-a720-68be65daa919-cert\") pod \"lws-controller-manager-6f6cfd7d7-s6cd2\" (UID: \"b920cc63-c662-4048-a720-68be65daa919\") " pod="openshift-lws-operator/lws-controller-manager-6f6cfd7d7-s6cd2"
May 25 13:20:57.129794 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:20:57.129742    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-8d6zz\" (UniqueName: \"kubernetes.io/projected/b920cc63-c662-4048-a720-68be65daa919-kube-api-access-8d6zz\") pod \"lws-controller-manager-6f6cfd7d7-s6cd2\" (UID: \"b920cc63-c662-4048-a720-68be65daa919\") " pod="openshift-lws-operator/lws-controller-manager-6f6cfd7d7-s6cd2"
May 25 13:20:57.129991 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:20:57.129802    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/b920cc63-c662-4048-a720-68be65daa919-manager-config\") pod \"lws-controller-manager-6f6cfd7d7-s6cd2\" (UID: \"b920cc63-c662-4048-a720-68be65daa919\") " pod="openshift-lws-operator/lws-controller-manager-6f6cfd7d7-s6cd2"
May 25 13:20:57.129991 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:20:57.129848    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/b920cc63-c662-4048-a720-68be65daa919-metrics-cert\") pod \"lws-controller-manager-6f6cfd7d7-s6cd2\" (UID: \"b920cc63-c662-4048-a720-68be65daa919\") " pod="openshift-lws-operator/lws-controller-manager-6f6cfd7d7-s6cd2"
May 25 13:20:57.130492 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:20:57.130469    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/b920cc63-c662-4048-a720-68be65daa919-manager-config\") pod \"lws-controller-manager-6f6cfd7d7-s6cd2\" (UID: \"b920cc63-c662-4048-a720-68be65daa919\") " pod="openshift-lws-operator/lws-controller-manager-6f6cfd7d7-s6cd2"
May 25 13:20:57.132309 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:20:57.132288    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b920cc63-c662-4048-a720-68be65daa919-cert\") pod \"lws-controller-manager-6f6cfd7d7-s6cd2\" (UID: \"b920cc63-c662-4048-a720-68be65daa919\") " pod="openshift-lws-operator/lws-controller-manager-6f6cfd7d7-s6cd2"
May 25 13:20:57.132416 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:20:57.132318    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/b920cc63-c662-4048-a720-68be65daa919-metrics-cert\") pod \"lws-controller-manager-6f6cfd7d7-s6cd2\" (UID: \"b920cc63-c662-4048-a720-68be65daa919\") " pod="openshift-lws-operator/lws-controller-manager-6f6cfd7d7-s6cd2"
May 25 13:20:57.143324 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:20:57.143300    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8d6zz\" (UniqueName: \"kubernetes.io/projected/b920cc63-c662-4048-a720-68be65daa919-kube-api-access-8d6zz\") pod \"lws-controller-manager-6f6cfd7d7-s6cd2\" (UID: \"b920cc63-c662-4048-a720-68be65daa919\") " pod="openshift-lws-operator/lws-controller-manager-6f6cfd7d7-s6cd2"
May 25 13:20:57.179493 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:20:57.179443    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6f6cfd7d7-s6cd2"
May 25 13:20:57.338675 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:20:57.338648    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6f6cfd7d7-s6cd2"]
May 25 13:20:57.340910 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:20:57.340882    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb920cc63_c662_4048_a720_68be65daa919.slice/crio-88f5a4ee7a3a729971f64488b282e7c6be090c5fb05f10fc377a3528de9eec6b WatchSource:0}: Error finding container 88f5a4ee7a3a729971f64488b282e7c6be090c5fb05f10fc377a3528de9eec6b: Status 404 returned error can't find the container with id 88f5a4ee7a3a729971f64488b282e7c6be090c5fb05f10fc377a3528de9eec6b
May 25 13:20:58.123409 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:20:58.123322    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6f6cfd7d7-s6cd2" event={"ID":"b920cc63-c662-4048-a720-68be65daa919","Type":"ContainerStarted","Data":"88f5a4ee7a3a729971f64488b282e7c6be090c5fb05f10fc377a3528de9eec6b"}
May 25 13:21:00.132609 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:21:00.132576    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6f6cfd7d7-s6cd2" event={"ID":"b920cc63-c662-4048-a720-68be65daa919","Type":"ContainerStarted","Data":"68f4e834c279d7f791b95a4cf6feba6c8b814fad6c7e2a86faa20e37970f8dda"}
May 25 13:21:00.132987 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:21:00.132639    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-6f6cfd7d7-s6cd2"
May 25 13:21:11.139155 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:21:11.139121    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-6f6cfd7d7-s6cd2"
May 25 13:21:11.203722 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:21:11.203670    2557 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-6f6cfd7d7-s6cd2" podStartSLOduration=12.76026469 podStartE2EDuration="15.203655835s" podCreationTimestamp="2026-05-25 13:20:56 +0000 UTC" firstStartedPulling="2026-05-25 13:20:57.342537122 +0000 UTC m=+909.805553257" lastFinishedPulling="2026-05-25 13:20:59.785928254 +0000 UTC m=+912.248944402" observedRunningTime="2026-05-25 13:21:00.158331592 +0000 UTC m=+912.621347748" watchObservedRunningTime="2026-05-25 13:21:11.203655835 +0000 UTC m=+923.666671991"
May 25 13:21:32.277017 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:21:32.276982    2557 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-c7fb4c8d5-5nd5w"]
May 25 13:21:32.279577 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:21:32.279558    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-c7fb4c8d5-5nd5w"
May 25 13:21:32.282911 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:21:32.282885    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
May 25 13:21:32.284126 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:21:32.284100    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
May 25 13:21:32.284126 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:21:32.284100    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-jtnwh\""
May 25 13:21:32.297495 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:21:32.297461    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-c7fb4c8d5-5nd5w"]
May 25 13:21:32.346249 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:21:32.346208    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwws2\" (UniqueName: \"kubernetes.io/projected/2cc8c2ee-258a-4fe8-ae11-a37e414c1266-kube-api-access-kwws2\") pod \"limitador-operator-controller-manager-c7fb4c8d5-5nd5w\" (UID: \"2cc8c2ee-258a-4fe8-ae11-a37e414c1266\") " pod="kuadrant-system/limitador-operator-controller-manager-c7fb4c8d5-5nd5w"
May 25 13:21:32.447726 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:21:32.447688    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-kwws2\" (UniqueName: \"kubernetes.io/projected/2cc8c2ee-258a-4fe8-ae11-a37e414c1266-kube-api-access-kwws2\") pod \"limitador-operator-controller-manager-c7fb4c8d5-5nd5w\" (UID: \"2cc8c2ee-258a-4fe8-ae11-a37e414c1266\") " pod="kuadrant-system/limitador-operator-controller-manager-c7fb4c8d5-5nd5w"
May 25 13:21:32.457770 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:21:32.457739    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwws2\" (UniqueName: \"kubernetes.io/projected/2cc8c2ee-258a-4fe8-ae11-a37e414c1266-kube-api-access-kwws2\") pod \"limitador-operator-controller-manager-c7fb4c8d5-5nd5w\" (UID: \"2cc8c2ee-258a-4fe8-ae11-a37e414c1266\") " pod="kuadrant-system/limitador-operator-controller-manager-c7fb4c8d5-5nd5w"
May 25 13:21:32.590187 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:21:32.590144    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-c7fb4c8d5-5nd5w"
May 25 13:21:32.724308 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:21:32.724280    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-c7fb4c8d5-5nd5w"]
May 25 13:21:32.727153 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:21:32.727119    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2cc8c2ee_258a_4fe8_ae11_a37e414c1266.slice/crio-a7c9ab004f446c30f1dc075ca375b2cb002c69f7f994b12a5f968bd7be1a3fb5 WatchSource:0}: Error finding container a7c9ab004f446c30f1dc075ca375b2cb002c69f7f994b12a5f968bd7be1a3fb5: Status 404 returned error can't find the container with id a7c9ab004f446c30f1dc075ca375b2cb002c69f7f994b12a5f968bd7be1a3fb5
May 25 13:21:33.237241 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:21:33.237204    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-c7fb4c8d5-5nd5w" event={"ID":"2cc8c2ee-258a-4fe8-ae11-a37e414c1266","Type":"ContainerStarted","Data":"a7c9ab004f446c30f1dc075ca375b2cb002c69f7f994b12a5f968bd7be1a3fb5"}
May 25 13:21:36.249130 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:21:36.249091    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-c7fb4c8d5-5nd5w" event={"ID":"2cc8c2ee-258a-4fe8-ae11-a37e414c1266","Type":"ContainerStarted","Data":"027d9354c90632ef80c0893f7f83afbbe85d0c9f7ec1e721e8a72ccab2911805"}
May 25 13:21:36.249588 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:21:36.249211    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-c7fb4c8d5-5nd5w"
May 25 13:21:36.275529 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:21:36.275471    2557 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-c7fb4c8d5-5nd5w" podStartSLOduration=1.786176771 podStartE2EDuration="4.275453421s" podCreationTimestamp="2026-05-25 13:21:32 +0000 UTC" firstStartedPulling="2026-05-25 13:21:32.729238051 +0000 UTC m=+945.192254186" lastFinishedPulling="2026-05-25 13:21:35.218514701 +0000 UTC m=+947.681530836" observedRunningTime="2026-05-25 13:21:36.274619725 +0000 UTC m=+948.737635882" watchObservedRunningTime="2026-05-25 13:21:36.275453421 +0000 UTC m=+948.738469578"
May 25 13:21:47.255119 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:21:47.255087    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-c7fb4c8d5-5nd5w"
May 25 13:22:34.552288 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:22:34.552251    2557 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-64c8f475fb-nzvzz"]
May 25 13:22:34.554581 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:22:34.554564    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c8f475fb-nzvzz"
May 25 13:22:34.557277 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:22:34.557252    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
May 25 13:22:34.557486 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:22:34.557468    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-p5cd7\""
May 25 13:22:34.564374 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:22:34.564333    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c8f475fb-nzvzz"]
May 25 13:22:34.648058 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:22:34.648027    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c8f475fb-nzvzz"]
May 25 13:22:34.683744 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:22:34.683708    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8nljq\" (UniqueName: \"kubernetes.io/projected/e662873d-a431-4c88-bbe4-79ee4aa68b45-kube-api-access-8nljq\") pod \"limitador-limitador-64c8f475fb-nzvzz\" (UID: \"e662873d-a431-4c88-bbe4-79ee4aa68b45\") " pod="kuadrant-system/limitador-limitador-64c8f475fb-nzvzz"
May 25 13:22:34.683895 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:22:34.683752    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/e662873d-a431-4c88-bbe4-79ee4aa68b45-config-file\") pod \"limitador-limitador-64c8f475fb-nzvzz\" (UID: \"e662873d-a431-4c88-bbe4-79ee4aa68b45\") " pod="kuadrant-system/limitador-limitador-64c8f475fb-nzvzz"
May 25 13:22:34.784813 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:22:34.784778    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-8nljq\" (UniqueName: \"kubernetes.io/projected/e662873d-a431-4c88-bbe4-79ee4aa68b45-kube-api-access-8nljq\") pod \"limitador-limitador-64c8f475fb-nzvzz\" (UID: \"e662873d-a431-4c88-bbe4-79ee4aa68b45\") " pod="kuadrant-system/limitador-limitador-64c8f475fb-nzvzz"
May 25 13:22:34.784978 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:22:34.784819    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/e662873d-a431-4c88-bbe4-79ee4aa68b45-config-file\") pod \"limitador-limitador-64c8f475fb-nzvzz\" (UID: \"e662873d-a431-4c88-bbe4-79ee4aa68b45\") " pod="kuadrant-system/limitador-limitador-64c8f475fb-nzvzz"
May 25 13:22:34.785432 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:22:34.785414    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/e662873d-a431-4c88-bbe4-79ee4aa68b45-config-file\") pod \"limitador-limitador-64c8f475fb-nzvzz\" (UID: \"e662873d-a431-4c88-bbe4-79ee4aa68b45\") " pod="kuadrant-system/limitador-limitador-64c8f475fb-nzvzz"
May 25 13:22:34.792589 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:22:34.792558    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8nljq\" (UniqueName: \"kubernetes.io/projected/e662873d-a431-4c88-bbe4-79ee4aa68b45-kube-api-access-8nljq\") pod \"limitador-limitador-64c8f475fb-nzvzz\" (UID: \"e662873d-a431-4c88-bbe4-79ee4aa68b45\") " pod="kuadrant-system/limitador-limitador-64c8f475fb-nzvzz"
May 25 13:22:34.864829 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:22:34.864736    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c8f475fb-nzvzz"
May 25 13:22:35.005862 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:22:35.005831    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c8f475fb-nzvzz"]
May 25 13:22:35.008605 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:22:35.008563    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode662873d_a431_4c88_bbe4_79ee4aa68b45.slice/crio-4248d96945b13d302b5aa320ff9479189e74c7a4e2f9e543b7f3280ad8ea3300 WatchSource:0}: Error finding container 4248d96945b13d302b5aa320ff9479189e74c7a4e2f9e543b7f3280ad8ea3300: Status 404 returned error can't find the container with id 4248d96945b13d302b5aa320ff9479189e74c7a4e2f9e543b7f3280ad8ea3300
May 25 13:22:35.381592 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:22:35.381550    2557 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-674b59b84c-zjd8x"]
May 25 13:22:35.384906 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:22:35.384884    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-674b59b84c-zjd8x"
May 25 13:22:35.388633 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:22:35.388610    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-btmv6\""
May 25 13:22:35.392051 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:22:35.392022    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-674b59b84c-zjd8x"]
May 25 13:22:35.435208 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:22:35.435170    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c8f475fb-nzvzz" event={"ID":"e662873d-a431-4c88-bbe4-79ee4aa68b45","Type":"ContainerStarted","Data":"4248d96945b13d302b5aa320ff9479189e74c7a4e2f9e543b7f3280ad8ea3300"}
May 25 13:22:35.492477 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:22:35.492442    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkr75\" (UniqueName: \"kubernetes.io/projected/3f3d66c5-daed-4a9e-8fff-ad0501110702-kube-api-access-jkr75\") pod \"authorino-674b59b84c-zjd8x\" (UID: \"3f3d66c5-daed-4a9e-8fff-ad0501110702\") " pod="kuadrant-system/authorino-674b59b84c-zjd8x"
May 25 13:22:35.593577 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:22:35.593541    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-jkr75\" (UniqueName: \"kubernetes.io/projected/3f3d66c5-daed-4a9e-8fff-ad0501110702-kube-api-access-jkr75\") pod \"authorino-674b59b84c-zjd8x\" (UID: \"3f3d66c5-daed-4a9e-8fff-ad0501110702\") " pod="kuadrant-system/authorino-674b59b84c-zjd8x"
May 25 13:22:35.602548 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:22:35.602520    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkr75\" (UniqueName: \"kubernetes.io/projected/3f3d66c5-daed-4a9e-8fff-ad0501110702-kube-api-access-jkr75\") pod \"authorino-674b59b84c-zjd8x\" (UID: \"3f3d66c5-daed-4a9e-8fff-ad0501110702\") " pod="kuadrant-system/authorino-674b59b84c-zjd8x"
May 25 13:22:35.696152 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:22:35.696061    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-674b59b84c-zjd8x"
May 25 13:22:35.867733 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:22:35.867700    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-674b59b84c-zjd8x"]
May 25 13:22:35.871939 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:22:35.870628    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3f3d66c5_daed_4a9e_8fff_ad0501110702.slice/crio-5ccbd123d1a685cfc06272bd2c33dce028b054db5970b8bb35e2647ee9fa3550 WatchSource:0}: Error finding container 5ccbd123d1a685cfc06272bd2c33dce028b054db5970b8bb35e2647ee9fa3550: Status 404 returned error can't find the container with id 5ccbd123d1a685cfc06272bd2c33dce028b054db5970b8bb35e2647ee9fa3550
May 25 13:22:36.451852 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:22:36.451817    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-674b59b84c-zjd8x" event={"ID":"3f3d66c5-daed-4a9e-8fff-ad0501110702","Type":"ContainerStarted","Data":"5ccbd123d1a685cfc06272bd2c33dce028b054db5970b8bb35e2647ee9fa3550"}
May 25 13:22:38.029341 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:22:38.029309    2557 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-674b59b84c-zjd8x"]
May 25 13:22:40.469331 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:22:40.469293    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c8f475fb-nzvzz" event={"ID":"e662873d-a431-4c88-bbe4-79ee4aa68b45","Type":"ContainerStarted","Data":"f6ebcb79e4c2307f57944af09481754dd8d9cc203fc9becd78af79eb8e7640a0"}
May 25 13:22:40.469835 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:22:40.469403    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-64c8f475fb-nzvzz"
May 25 13:22:40.470611 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:22:40.470587    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-674b59b84c-zjd8x" event={"ID":"3f3d66c5-daed-4a9e-8fff-ad0501110702","Type":"ContainerStarted","Data":"a2ac44db97e77f8d5cb4f204c7936a8bdb13dcebff05a98a41a122f52c0bb46a"}
May 25 13:22:40.470677 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:22:40.470660    2557 kuberuntime_container.go:864] "Killing container with a grace period" pod="kuadrant-system/authorino-674b59b84c-zjd8x" podUID="3f3d66c5-daed-4a9e-8fff-ad0501110702" containerName="authorino" containerID="cri-o://a2ac44db97e77f8d5cb4f204c7936a8bdb13dcebff05a98a41a122f52c0bb46a" gracePeriod=30
May 25 13:22:40.486309 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:22:40.486247    2557 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-64c8f475fb-nzvzz" podStartSLOduration=1.585785633 podStartE2EDuration="6.486226737s" podCreationTimestamp="2026-05-25 13:22:34 +0000 UTC" firstStartedPulling="2026-05-25 13:22:35.010465722 +0000 UTC m=+1007.473481860" lastFinishedPulling="2026-05-25 13:22:39.910906819 +0000 UTC m=+1012.373922964" observedRunningTime="2026-05-25 13:22:40.48564788 +0000 UTC m=+1012.948664051" watchObservedRunningTime="2026-05-25 13:22:40.486226737 +0000 UTC m=+1012.949242896"
May 25 13:22:40.501338 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:22:40.501284    2557 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kuadrant-system/authorino-674b59b84c-zjd8x" podStartSLOduration=1.452688063 podStartE2EDuration="5.501270327s" podCreationTimestamp="2026-05-25 13:22:35 +0000 UTC" firstStartedPulling="2026-05-25 13:22:35.87248955 +0000 UTC m=+1008.335505689" lastFinishedPulling="2026-05-25 13:22:39.921071626 +0000 UTC m=+1012.384087953" observedRunningTime="2026-05-25 13:22:40.500019462 +0000 UTC m=+1012.963035628" watchObservedRunningTime="2026-05-25 13:22:40.501270327 +0000 UTC m=+1012.964286482"
May 25 13:22:40.713823 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:22:40.713798    2557 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-674b59b84c-zjd8x"
May 25 13:22:40.845453 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:22:40.845419    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jkr75\" (UniqueName: \"kubernetes.io/projected/3f3d66c5-daed-4a9e-8fff-ad0501110702-kube-api-access-jkr75\") pod \"3f3d66c5-daed-4a9e-8fff-ad0501110702\" (UID: \"3f3d66c5-daed-4a9e-8fff-ad0501110702\") "
May 25 13:22:40.847638 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:22:40.847609    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f3d66c5-daed-4a9e-8fff-ad0501110702-kube-api-access-jkr75" (OuterVolumeSpecName: "kube-api-access-jkr75") pod "3f3d66c5-daed-4a9e-8fff-ad0501110702" (UID: "3f3d66c5-daed-4a9e-8fff-ad0501110702"). InnerVolumeSpecName "kube-api-access-jkr75". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 13:22:40.947177 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:22:40.947135    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jkr75\" (UniqueName: \"kubernetes.io/projected/3f3d66c5-daed-4a9e-8fff-ad0501110702-kube-api-access-jkr75\") on node \"ip-10-0-143-235.ec2.internal\" DevicePath \"\""
May 25 13:22:41.475149 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:22:41.475111    2557 generic.go:358] "Generic (PLEG): container finished" podID="3f3d66c5-daed-4a9e-8fff-ad0501110702" containerID="a2ac44db97e77f8d5cb4f204c7936a8bdb13dcebff05a98a41a122f52c0bb46a" exitCode=0
May 25 13:22:41.475634 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:22:41.475167    2557 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-674b59b84c-zjd8x"
May 25 13:22:41.475634 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:22:41.475194    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-674b59b84c-zjd8x" event={"ID":"3f3d66c5-daed-4a9e-8fff-ad0501110702","Type":"ContainerDied","Data":"a2ac44db97e77f8d5cb4f204c7936a8bdb13dcebff05a98a41a122f52c0bb46a"}
May 25 13:22:41.475634 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:22:41.475229    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-674b59b84c-zjd8x" event={"ID":"3f3d66c5-daed-4a9e-8fff-ad0501110702","Type":"ContainerDied","Data":"5ccbd123d1a685cfc06272bd2c33dce028b054db5970b8bb35e2647ee9fa3550"}
May 25 13:22:41.475634 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:22:41.475244    2557 scope.go:117] "RemoveContainer" containerID="a2ac44db97e77f8d5cb4f204c7936a8bdb13dcebff05a98a41a122f52c0bb46a"
May 25 13:22:41.483392 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:22:41.483363    2557 scope.go:117] "RemoveContainer" containerID="a2ac44db97e77f8d5cb4f204c7936a8bdb13dcebff05a98a41a122f52c0bb46a"
May 25 13:22:41.483659 ip-10-0-143-235 kubenswrapper[2557]: E0525 13:22:41.483634    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2ac44db97e77f8d5cb4f204c7936a8bdb13dcebff05a98a41a122f52c0bb46a\": container with ID starting with a2ac44db97e77f8d5cb4f204c7936a8bdb13dcebff05a98a41a122f52c0bb46a not found: ID does not exist" containerID="a2ac44db97e77f8d5cb4f204c7936a8bdb13dcebff05a98a41a122f52c0bb46a"
May 25 13:22:41.483736 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:22:41.483663    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2ac44db97e77f8d5cb4f204c7936a8bdb13dcebff05a98a41a122f52c0bb46a"} err="failed to get container status \"a2ac44db97e77f8d5cb4f204c7936a8bdb13dcebff05a98a41a122f52c0bb46a\": rpc error: code = NotFound desc = could not find container \"a2ac44db97e77f8d5cb4f204c7936a8bdb13dcebff05a98a41a122f52c0bb46a\": container with ID starting with a2ac44db97e77f8d5cb4f204c7936a8bdb13dcebff05a98a41a122f52c0bb46a not found: ID does not exist"
May 25 13:22:41.497399 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:22:41.497368    2557 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-674b59b84c-zjd8x"]
May 25 13:22:41.500631 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:22:41.500606    2557 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-674b59b84c-zjd8x"]
May 25 13:22:42.150401 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:22:42.150368    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3f3d66c5-daed-4a9e-8fff-ad0501110702" path="/var/lib/kubelet/pods/3f3d66c5-daed-4a9e-8fff-ad0501110702/volumes"
May 25 13:22:47.530402 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:22:47.530360    2557 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c8f475fb-nzvzz"]
May 25 13:22:47.530815 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:22:47.530591    2557 kuberuntime_container.go:864] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-64c8f475fb-nzvzz" podUID="e662873d-a431-4c88-bbe4-79ee4aa68b45" containerName="limitador" containerID="cri-o://f6ebcb79e4c2307f57944af09481754dd8d9cc203fc9becd78af79eb8e7640a0" gracePeriod=30
May 25 13:22:47.531286 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:22:47.531224    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-64c8f475fb-nzvzz"
May 25 13:22:48.075232 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:22:48.075208    2557 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c8f475fb-nzvzz"
May 25 13:22:48.210927 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:22:48.210874    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8nljq\" (UniqueName: \"kubernetes.io/projected/e662873d-a431-4c88-bbe4-79ee4aa68b45-kube-api-access-8nljq\") pod \"e662873d-a431-4c88-bbe4-79ee4aa68b45\" (UID: \"e662873d-a431-4c88-bbe4-79ee4aa68b45\") "
May 25 13:22:48.210927 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:22:48.210928    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/e662873d-a431-4c88-bbe4-79ee4aa68b45-config-file\") pod \"e662873d-a431-4c88-bbe4-79ee4aa68b45\" (UID: \"e662873d-a431-4c88-bbe4-79ee4aa68b45\") "
May 25 13:22:48.211278 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:22:48.211252    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e662873d-a431-4c88-bbe4-79ee4aa68b45-config-file" (OuterVolumeSpecName: "config-file") pod "e662873d-a431-4c88-bbe4-79ee4aa68b45" (UID: "e662873d-a431-4c88-bbe4-79ee4aa68b45"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 13:22:48.213016 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:22:48.212991    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e662873d-a431-4c88-bbe4-79ee4aa68b45-kube-api-access-8nljq" (OuterVolumeSpecName: "kube-api-access-8nljq") pod "e662873d-a431-4c88-bbe4-79ee4aa68b45" (UID: "e662873d-a431-4c88-bbe4-79ee4aa68b45"). InnerVolumeSpecName "kube-api-access-8nljq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 13:22:48.312077 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:22:48.312037    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8nljq\" (UniqueName: \"kubernetes.io/projected/e662873d-a431-4c88-bbe4-79ee4aa68b45-kube-api-access-8nljq\") on node \"ip-10-0-143-235.ec2.internal\" DevicePath \"\""
May 25 13:22:48.312077 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:22:48.312071    2557 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/e662873d-a431-4c88-bbe4-79ee4aa68b45-config-file\") on node \"ip-10-0-143-235.ec2.internal\" DevicePath \"\""
May 25 13:22:48.501406 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:22:48.501298    2557 generic.go:358] "Generic (PLEG): container finished" podID="e662873d-a431-4c88-bbe4-79ee4aa68b45" containerID="f6ebcb79e4c2307f57944af09481754dd8d9cc203fc9becd78af79eb8e7640a0" exitCode=0
May 25 13:22:48.501406 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:22:48.501389    2557 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c8f475fb-nzvzz"
May 25 13:22:48.501576 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:22:48.501390    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c8f475fb-nzvzz" event={"ID":"e662873d-a431-4c88-bbe4-79ee4aa68b45","Type":"ContainerDied","Data":"f6ebcb79e4c2307f57944af09481754dd8d9cc203fc9becd78af79eb8e7640a0"}
May 25 13:22:48.501576 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:22:48.501500    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c8f475fb-nzvzz" event={"ID":"e662873d-a431-4c88-bbe4-79ee4aa68b45","Type":"ContainerDied","Data":"4248d96945b13d302b5aa320ff9479189e74c7a4e2f9e543b7f3280ad8ea3300"}
May 25 13:22:48.501576 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:22:48.501515    2557 scope.go:117] "RemoveContainer" containerID="f6ebcb79e4c2307f57944af09481754dd8d9cc203fc9becd78af79eb8e7640a0"
May 25 13:22:48.509578 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:22:48.509561    2557 scope.go:117] "RemoveContainer" containerID="f6ebcb79e4c2307f57944af09481754dd8d9cc203fc9becd78af79eb8e7640a0"
May 25 13:22:48.509879 ip-10-0-143-235 kubenswrapper[2557]: E0525 13:22:48.509842    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6ebcb79e4c2307f57944af09481754dd8d9cc203fc9becd78af79eb8e7640a0\": container with ID starting with f6ebcb79e4c2307f57944af09481754dd8d9cc203fc9becd78af79eb8e7640a0 not found: ID does not exist" containerID="f6ebcb79e4c2307f57944af09481754dd8d9cc203fc9becd78af79eb8e7640a0"
May 25 13:22:48.509948 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:22:48.509888    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6ebcb79e4c2307f57944af09481754dd8d9cc203fc9becd78af79eb8e7640a0"} err="failed to get container status \"f6ebcb79e4c2307f57944af09481754dd8d9cc203fc9becd78af79eb8e7640a0\": rpc error: code = NotFound desc = could not find container \"f6ebcb79e4c2307f57944af09481754dd8d9cc203fc9becd78af79eb8e7640a0\": container with ID starting with f6ebcb79e4c2307f57944af09481754dd8d9cc203fc9becd78af79eb8e7640a0 not found: ID does not exist"
May 25 13:22:48.523713 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:22:48.523681    2557 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c8f475fb-nzvzz"]
May 25 13:22:48.527494 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:22:48.527469    2557 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-64c8f475fb-nzvzz"]
May 25 13:22:50.150295 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:22:50.150251    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e662873d-a431-4c88-bbe4-79ee4aa68b45" path="/var/lib/kubelet/pods/e662873d-a431-4c88-bbe4-79ee4aa68b45/volumes"
May 25 13:22:56.368646 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:22:56.368608    2557 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-68bd676465-8zsn9"]
May 25 13:22:56.369014 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:22:56.368957    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e662873d-a431-4c88-bbe4-79ee4aa68b45" containerName="limitador"
May 25 13:22:56.369014 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:22:56.368968    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="e662873d-a431-4c88-bbe4-79ee4aa68b45" containerName="limitador"
May 25 13:22:56.369014 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:22:56.368977    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f3d66c5-daed-4a9e-8fff-ad0501110702" containerName="authorino"
May 25 13:22:56.369014 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:22:56.368984    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f3d66c5-daed-4a9e-8fff-ad0501110702" containerName="authorino"
May 25 13:22:56.369138 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:22:56.369052    2557 memory_manager.go:356] "RemoveStaleState removing state" podUID="e662873d-a431-4c88-bbe4-79ee4aa68b45" containerName="limitador"
May 25 13:22:56.369138 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:22:56.369064    2557 memory_manager.go:356] "RemoveStaleState removing state" podUID="3f3d66c5-daed-4a9e-8fff-ad0501110702" containerName="authorino"
May 25 13:22:56.373342 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:22:56.373324    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-68bd676465-8zsn9"
May 25 13:22:56.377435 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:22:56.377407    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
May 25 13:22:56.377435 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:22:56.377425    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-btmv6\""
May 25 13:22:56.379850 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:22:56.379826    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-68bd676465-8zsn9"]
May 25 13:22:56.488051 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:22:56.488020    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/3c78c0b9-30ff-420b-8fed-1e5537f4356d-tls-cert\") pod \"authorino-68bd676465-8zsn9\" (UID: \"3c78c0b9-30ff-420b-8fed-1e5537f4356d\") " pod="kuadrant-system/authorino-68bd676465-8zsn9"
May 25 13:22:56.488199 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:22:56.488066    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7t8xx\" (UniqueName: \"kubernetes.io/projected/3c78c0b9-30ff-420b-8fed-1e5537f4356d-kube-api-access-7t8xx\") pod \"authorino-68bd676465-8zsn9\" (UID: \"3c78c0b9-30ff-420b-8fed-1e5537f4356d\") " pod="kuadrant-system/authorino-68bd676465-8zsn9"
May 25 13:22:56.588983 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:22:56.588942    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/3c78c0b9-30ff-420b-8fed-1e5537f4356d-tls-cert\") pod \"authorino-68bd676465-8zsn9\" (UID: \"3c78c0b9-30ff-420b-8fed-1e5537f4356d\") " pod="kuadrant-system/authorino-68bd676465-8zsn9"
May 25 13:22:56.589160 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:22:56.588995    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-7t8xx\" (UniqueName: \"kubernetes.io/projected/3c78c0b9-30ff-420b-8fed-1e5537f4356d-kube-api-access-7t8xx\") pod \"authorino-68bd676465-8zsn9\" (UID: \"3c78c0b9-30ff-420b-8fed-1e5537f4356d\") " pod="kuadrant-system/authorino-68bd676465-8zsn9"
May 25 13:22:56.591430 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:22:56.591409    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/3c78c0b9-30ff-420b-8fed-1e5537f4356d-tls-cert\") pod \"authorino-68bd676465-8zsn9\" (UID: \"3c78c0b9-30ff-420b-8fed-1e5537f4356d\") " pod="kuadrant-system/authorino-68bd676465-8zsn9"
May 25 13:22:56.597982 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:22:56.597960    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7t8xx\" (UniqueName: \"kubernetes.io/projected/3c78c0b9-30ff-420b-8fed-1e5537f4356d-kube-api-access-7t8xx\") pod \"authorino-68bd676465-8zsn9\" (UID: \"3c78c0b9-30ff-420b-8fed-1e5537f4356d\") " pod="kuadrant-system/authorino-68bd676465-8zsn9"
May 25 13:22:56.683704 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:22:56.683606    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-68bd676465-8zsn9"
May 25 13:22:56.816752 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:22:56.816714    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-68bd676465-8zsn9"]
May 25 13:22:56.819549 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:22:56.819520    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3c78c0b9_30ff_420b_8fed_1e5537f4356d.slice/crio-217388093efb0ddbbd53dbbcee40421bba09d5029f6b6e4fc78d25149cc9621e WatchSource:0}: Error finding container 217388093efb0ddbbd53dbbcee40421bba09d5029f6b6e4fc78d25149cc9621e: Status 404 returned error can't find the container with id 217388093efb0ddbbd53dbbcee40421bba09d5029f6b6e4fc78d25149cc9621e
May 25 13:22:57.535461 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:22:57.535341    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-68bd676465-8zsn9" event={"ID":"3c78c0b9-30ff-420b-8fed-1e5537f4356d","Type":"ContainerStarted","Data":"8c9d8239ae1b1862735f3685ebfe5c867bb7c57aed95e0e0ec01e3b14cae0d12"}
May 25 13:22:57.535461 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:22:57.535410    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-68bd676465-8zsn9" event={"ID":"3c78c0b9-30ff-420b-8fed-1e5537f4356d","Type":"ContainerStarted","Data":"217388093efb0ddbbd53dbbcee40421bba09d5029f6b6e4fc78d25149cc9621e"}
May 25 13:22:57.551523 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:22:57.551469    2557 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kuadrant-system/authorino-68bd676465-8zsn9" podStartSLOduration=1.19069044 podStartE2EDuration="1.551453944s" podCreationTimestamp="2026-05-25 13:22:56 +0000 UTC" firstStartedPulling="2026-05-25 13:22:56.820796539 +0000 UTC m=+1029.283812673" lastFinishedPulling="2026-05-25 13:22:57.181560038 +0000 UTC m=+1029.644576177" observedRunningTime="2026-05-25 13:22:57.55106801 +0000 UTC m=+1030.014084161" watchObservedRunningTime="2026-05-25 13:22:57.551453944 +0000 UTC m=+1030.014470101"
May 25 13:23:16.064367 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:23:16.064321    2557 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-7cd9b78f5d-g8fjl"]
May 25 13:23:16.066941 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:23:16.066925    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-7cd9b78f5d-g8fjl"
May 25 13:23:16.070919 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:23:16.070894    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
May 25 13:23:16.070919 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:23:16.070906    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
May 25 13:23:16.071138 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:23:16.070966    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-controller-manager-dockercfg-j5h2k\""
May 25 13:23:16.071138 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:23:16.070898    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-webhook-server-cert\""
May 25 13:23:16.079075 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:23:16.079048    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-7cd9b78f5d-g8fjl"]
May 25 13:23:16.082074 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:23:16.082049    2557 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve/llmisvc-controller-manager-6644479599-4x2cl"]
May 25 13:23:16.084684 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:23:16.084664    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-6644479599-4x2cl"
May 25 13:23:16.087236 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:23:16.087218    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-webhook-server-cert\""
May 25 13:23:16.087374 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:23:16.087290    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-controller-manager-dockercfg-r9zrq\""
May 25 13:23:16.097483 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:23:16.097457    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-6644479599-4x2cl"]
May 25 13:23:16.150869 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:23:16.150829    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/639b620f-34a7-42f4-866e-f769f4d753c4-cert\") pod \"llmisvc-controller-manager-6644479599-4x2cl\" (UID: \"639b620f-34a7-42f4-866e-f769f4d753c4\") " pod="kserve/llmisvc-controller-manager-6644479599-4x2cl"
May 25 13:23:16.151045 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:23:16.150889    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2sc68\" (UniqueName: \"kubernetes.io/projected/e8f60531-4177-43d7-90e5-eb33298d910e-kube-api-access-2sc68\") pod \"kserve-controller-manager-7cd9b78f5d-g8fjl\" (UID: \"e8f60531-4177-43d7-90e5-eb33298d910e\") " pod="kserve/kserve-controller-manager-7cd9b78f5d-g8fjl"
May 25 13:23:16.151045 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:23:16.150915    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e8f60531-4177-43d7-90e5-eb33298d910e-cert\") pod \"kserve-controller-manager-7cd9b78f5d-g8fjl\" (UID: \"e8f60531-4177-43d7-90e5-eb33298d910e\") " pod="kserve/kserve-controller-manager-7cd9b78f5d-g8fjl"
May 25 13:23:16.151045 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:23:16.150934    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74tpc\" (UniqueName: \"kubernetes.io/projected/639b620f-34a7-42f4-866e-f769f4d753c4-kube-api-access-74tpc\") pod \"llmisvc-controller-manager-6644479599-4x2cl\" (UID: \"639b620f-34a7-42f4-866e-f769f4d753c4\") " pod="kserve/llmisvc-controller-manager-6644479599-4x2cl"
May 25 13:23:16.252191 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:23:16.252152    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-2sc68\" (UniqueName: \"kubernetes.io/projected/e8f60531-4177-43d7-90e5-eb33298d910e-kube-api-access-2sc68\") pod \"kserve-controller-manager-7cd9b78f5d-g8fjl\" (UID: \"e8f60531-4177-43d7-90e5-eb33298d910e\") " pod="kserve/kserve-controller-manager-7cd9b78f5d-g8fjl"
May 25 13:23:16.252444 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:23:16.252203    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e8f60531-4177-43d7-90e5-eb33298d910e-cert\") pod \"kserve-controller-manager-7cd9b78f5d-g8fjl\" (UID: \"e8f60531-4177-43d7-90e5-eb33298d910e\") " pod="kserve/kserve-controller-manager-7cd9b78f5d-g8fjl"
May 25 13:23:16.252444 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:23:16.252237    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-74tpc\" (UniqueName: \"kubernetes.io/projected/639b620f-34a7-42f4-866e-f769f4d753c4-kube-api-access-74tpc\") pod \"llmisvc-controller-manager-6644479599-4x2cl\" (UID: \"639b620f-34a7-42f4-866e-f769f4d753c4\") " pod="kserve/llmisvc-controller-manager-6644479599-4x2cl"
May 25 13:23:16.252444 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:23:16.252325    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/639b620f-34a7-42f4-866e-f769f4d753c4-cert\") pod \"llmisvc-controller-manager-6644479599-4x2cl\" (UID: \"639b620f-34a7-42f4-866e-f769f4d753c4\") " pod="kserve/llmisvc-controller-manager-6644479599-4x2cl"
May 25 13:23:16.255032 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:23:16.255004    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/639b620f-34a7-42f4-866e-f769f4d753c4-cert\") pod \"llmisvc-controller-manager-6644479599-4x2cl\" (UID: \"639b620f-34a7-42f4-866e-f769f4d753c4\") " pod="kserve/llmisvc-controller-manager-6644479599-4x2cl"
May 25 13:23:16.255136 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:23:16.255008    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e8f60531-4177-43d7-90e5-eb33298d910e-cert\") pod \"kserve-controller-manager-7cd9b78f5d-g8fjl\" (UID: \"e8f60531-4177-43d7-90e5-eb33298d910e\") " pod="kserve/kserve-controller-manager-7cd9b78f5d-g8fjl"
May 25 13:23:16.261441 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:23:16.261414    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2sc68\" (UniqueName: \"kubernetes.io/projected/e8f60531-4177-43d7-90e5-eb33298d910e-kube-api-access-2sc68\") pod \"kserve-controller-manager-7cd9b78f5d-g8fjl\" (UID: \"e8f60531-4177-43d7-90e5-eb33298d910e\") " pod="kserve/kserve-controller-manager-7cd9b78f5d-g8fjl"
May 25 13:23:16.261847 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:23:16.261831    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-74tpc\" (UniqueName: \"kubernetes.io/projected/639b620f-34a7-42f4-866e-f769f4d753c4-kube-api-access-74tpc\") pod \"llmisvc-controller-manager-6644479599-4x2cl\" (UID: \"639b620f-34a7-42f4-866e-f769f4d753c4\") " pod="kserve/llmisvc-controller-manager-6644479599-4x2cl"
May 25 13:23:16.379042 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:23:16.378950    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-7cd9b78f5d-g8fjl"
May 25 13:23:16.395772 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:23:16.395743    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-6644479599-4x2cl"
May 25 13:23:16.520506 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:23:16.520478    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-7cd9b78f5d-g8fjl"]
May 25 13:23:16.522607 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:23:16.522578    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8f60531_4177_43d7_90e5_eb33298d910e.slice/crio-50d4c3c0b3cc602248fed09badf12150c815b13afe89b18142d276ba249cf1a2 WatchSource:0}: Error finding container 50d4c3c0b3cc602248fed09badf12150c815b13afe89b18142d276ba249cf1a2: Status 404 returned error can't find the container with id 50d4c3c0b3cc602248fed09badf12150c815b13afe89b18142d276ba249cf1a2
May 25 13:23:16.543838 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:23:16.543815    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-6644479599-4x2cl"]
May 25 13:23:16.545604 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:23:16.545574    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod639b620f_34a7_42f4_866e_f769f4d753c4.slice/crio-066e315d275d89fc50246da0e71055d255b4de0535d62c5ea98d34b0554149c3 WatchSource:0}: Error finding container 066e315d275d89fc50246da0e71055d255b4de0535d62c5ea98d34b0554149c3: Status 404 returned error can't find the container with id 066e315d275d89fc50246da0e71055d255b4de0535d62c5ea98d34b0554149c3
May 25 13:23:16.615684 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:23:16.615647    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-6644479599-4x2cl" event={"ID":"639b620f-34a7-42f4-866e-f769f4d753c4","Type":"ContainerStarted","Data":"066e315d275d89fc50246da0e71055d255b4de0535d62c5ea98d34b0554149c3"}
May 25 13:23:16.616641 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:23:16.616613    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-7cd9b78f5d-g8fjl" event={"ID":"e8f60531-4177-43d7-90e5-eb33298d910e","Type":"ContainerStarted","Data":"50d4c3c0b3cc602248fed09badf12150c815b13afe89b18142d276ba249cf1a2"}
May 25 13:23:20.637502 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:23:20.637464    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-6644479599-4x2cl" event={"ID":"639b620f-34a7-42f4-866e-f769f4d753c4","Type":"ContainerStarted","Data":"f39a7593571cf57b9380e8c4fc7e1ca5fac968fe467ff05bd9bbf2b1c0592595"}
May 25 13:23:20.637994 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:23:20.637563    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-6644479599-4x2cl"
May 25 13:23:20.638833 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:23:20.638808    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-7cd9b78f5d-g8fjl" event={"ID":"e8f60531-4177-43d7-90e5-eb33298d910e","Type":"ContainerStarted","Data":"416502fb1f93226194e3ead69222ed2c9f50664e96b54d377bdc541ec5f913e8"}
May 25 13:23:20.638942 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:23:20.638921    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-7cd9b78f5d-g8fjl"
May 25 13:23:20.654593 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:23:20.654543    2557 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve/llmisvc-controller-manager-6644479599-4x2cl" podStartSLOduration=1.302772581 podStartE2EDuration="4.654529143s" podCreationTimestamp="2026-05-25 13:23:16 +0000 UTC" firstStartedPulling="2026-05-25 13:23:16.546978088 +0000 UTC m=+1049.009994223" lastFinishedPulling="2026-05-25 13:23:19.898734633 +0000 UTC m=+1052.361750785" observedRunningTime="2026-05-25 13:23:20.654211187 +0000 UTC m=+1053.117227345" watchObservedRunningTime="2026-05-25 13:23:20.654529143 +0000 UTC m=+1053.117545299"
May 25 13:23:20.672653 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:23:20.672598    2557 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve/kserve-controller-manager-7cd9b78f5d-g8fjl" podStartSLOduration=1.29802431 podStartE2EDuration="4.672584169s" podCreationTimestamp="2026-05-25 13:23:16 +0000 UTC" firstStartedPulling="2026-05-25 13:23:16.524009168 +0000 UTC m=+1048.987025302" lastFinishedPulling="2026-05-25 13:23:19.898569011 +0000 UTC m=+1052.361585161" observedRunningTime="2026-05-25 13:23:20.670709759 +0000 UTC m=+1053.133725927" watchObservedRunningTime="2026-05-25 13:23:20.672584169 +0000 UTC m=+1053.135600324"
May 25 13:23:51.645212 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:23:51.645175    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/llmisvc-controller-manager-6644479599-4x2cl"
May 25 13:23:51.648190 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:23:51.648165    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-7cd9b78f5d-g8fjl"
May 25 13:23:52.857103 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:23:52.857065    2557 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-7cd9b78f5d-g8fjl"]
May 25 13:23:52.857515 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:23:52.857290    2557 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve/kserve-controller-manager-7cd9b78f5d-g8fjl" podUID="e8f60531-4177-43d7-90e5-eb33298d910e" containerName="manager" containerID="cri-o://416502fb1f93226194e3ead69222ed2c9f50664e96b54d377bdc541ec5f913e8" gracePeriod=10
May 25 13:23:52.881774 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:23:52.881746    2557 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-7cd9b78f5d-b6kcm"]
May 25 13:23:52.886229 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:23:52.886207    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-7cd9b78f5d-b6kcm"
May 25 13:23:52.896500 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:23:52.896470    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-7cd9b78f5d-b6kcm"]
May 25 13:23:52.897796 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:23:52.897757    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fd65f3b4-c9f6-4141-b034-92cf6bafa8f3-cert\") pod \"kserve-controller-manager-7cd9b78f5d-b6kcm\" (UID: \"fd65f3b4-c9f6-4141-b034-92cf6bafa8f3\") " pod="kserve/kserve-controller-manager-7cd9b78f5d-b6kcm"
May 25 13:23:52.897936 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:23:52.897828    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwlmv\" (UniqueName: \"kubernetes.io/projected/fd65f3b4-c9f6-4141-b034-92cf6bafa8f3-kube-api-access-gwlmv\") pod \"kserve-controller-manager-7cd9b78f5d-b6kcm\" (UID: \"fd65f3b4-c9f6-4141-b034-92cf6bafa8f3\") " pod="kserve/kserve-controller-manager-7cd9b78f5d-b6kcm"
May 25 13:23:52.998459 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:23:52.998413    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-gwlmv\" (UniqueName: \"kubernetes.io/projected/fd65f3b4-c9f6-4141-b034-92cf6bafa8f3-kube-api-access-gwlmv\") pod \"kserve-controller-manager-7cd9b78f5d-b6kcm\" (UID: \"fd65f3b4-c9f6-4141-b034-92cf6bafa8f3\") " pod="kserve/kserve-controller-manager-7cd9b78f5d-b6kcm"
May 25 13:23:52.998628 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:23:52.998588    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fd65f3b4-c9f6-4141-b034-92cf6bafa8f3-cert\") pod \"kserve-controller-manager-7cd9b78f5d-b6kcm\" (UID: \"fd65f3b4-c9f6-4141-b034-92cf6bafa8f3\") " pod="kserve/kserve-controller-manager-7cd9b78f5d-b6kcm"
May 25 13:23:53.001698 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:23:53.001625    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fd65f3b4-c9f6-4141-b034-92cf6bafa8f3-cert\") pod \"kserve-controller-manager-7cd9b78f5d-b6kcm\" (UID: \"fd65f3b4-c9f6-4141-b034-92cf6bafa8f3\") " pod="kserve/kserve-controller-manager-7cd9b78f5d-b6kcm"
May 25 13:23:53.019107 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:23:53.019073    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwlmv\" (UniqueName: \"kubernetes.io/projected/fd65f3b4-c9f6-4141-b034-92cf6bafa8f3-kube-api-access-gwlmv\") pod \"kserve-controller-manager-7cd9b78f5d-b6kcm\" (UID: \"fd65f3b4-c9f6-4141-b034-92cf6bafa8f3\") " pod="kserve/kserve-controller-manager-7cd9b78f5d-b6kcm"
May 25 13:23:53.123249 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:23:53.123220    2557 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-7cd9b78f5d-g8fjl"
May 25 13:23:53.200268 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:23:53.200234    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2sc68\" (UniqueName: \"kubernetes.io/projected/e8f60531-4177-43d7-90e5-eb33298d910e-kube-api-access-2sc68\") pod \"e8f60531-4177-43d7-90e5-eb33298d910e\" (UID: \"e8f60531-4177-43d7-90e5-eb33298d910e\") "
May 25 13:23:53.200477 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:23:53.200288    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e8f60531-4177-43d7-90e5-eb33298d910e-cert\") pod \"e8f60531-4177-43d7-90e5-eb33298d910e\" (UID: \"e8f60531-4177-43d7-90e5-eb33298d910e\") "
May 25 13:23:53.202594 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:23:53.202569    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8f60531-4177-43d7-90e5-eb33298d910e-cert" (OuterVolumeSpecName: "cert") pod "e8f60531-4177-43d7-90e5-eb33298d910e" (UID: "e8f60531-4177-43d7-90e5-eb33298d910e"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 13:23:53.202694 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:23:53.202593    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8f60531-4177-43d7-90e5-eb33298d910e-kube-api-access-2sc68" (OuterVolumeSpecName: "kube-api-access-2sc68") pod "e8f60531-4177-43d7-90e5-eb33298d910e" (UID: "e8f60531-4177-43d7-90e5-eb33298d910e"). InnerVolumeSpecName "kube-api-access-2sc68". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 13:23:53.247804 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:23:53.247769    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-7cd9b78f5d-b6kcm"
May 25 13:23:53.301976 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:23:53.301936    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2sc68\" (UniqueName: \"kubernetes.io/projected/e8f60531-4177-43d7-90e5-eb33298d910e-kube-api-access-2sc68\") on node \"ip-10-0-143-235.ec2.internal\" DevicePath \"\""
May 25 13:23:53.301976 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:23:53.301978    2557 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e8f60531-4177-43d7-90e5-eb33298d910e-cert\") on node \"ip-10-0-143-235.ec2.internal\" DevicePath \"\""
May 25 13:23:53.381434 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:23:53.381407    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-7cd9b78f5d-b6kcm"]
May 25 13:23:53.383680 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:23:53.383653    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd65f3b4_c9f6_4141_b034_92cf6bafa8f3.slice/crio-a5bfa026707264a90bd189184936494b3791d0f13b1e2d63ebf74b19fe7c7122 WatchSource:0}: Error finding container a5bfa026707264a90bd189184936494b3791d0f13b1e2d63ebf74b19fe7c7122: Status 404 returned error can't find the container with id a5bfa026707264a90bd189184936494b3791d0f13b1e2d63ebf74b19fe7c7122
May 25 13:23:53.756135 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:23:53.756040    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-7cd9b78f5d-b6kcm" event={"ID":"fd65f3b4-c9f6-4141-b034-92cf6bafa8f3","Type":"ContainerStarted","Data":"a5bfa026707264a90bd189184936494b3791d0f13b1e2d63ebf74b19fe7c7122"}
May 25 13:23:53.757047 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:23:53.757025    2557 generic.go:358] "Generic (PLEG): container finished" podID="e8f60531-4177-43d7-90e5-eb33298d910e" containerID="416502fb1f93226194e3ead69222ed2c9f50664e96b54d377bdc541ec5f913e8" exitCode=0
May 25 13:23:53.757150 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:23:53.757074    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-7cd9b78f5d-g8fjl" event={"ID":"e8f60531-4177-43d7-90e5-eb33298d910e","Type":"ContainerDied","Data":"416502fb1f93226194e3ead69222ed2c9f50664e96b54d377bdc541ec5f913e8"}
May 25 13:23:53.757150 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:23:53.757082    2557 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-7cd9b78f5d-g8fjl"
May 25 13:23:53.757150 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:23:53.757091    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-7cd9b78f5d-g8fjl" event={"ID":"e8f60531-4177-43d7-90e5-eb33298d910e","Type":"ContainerDied","Data":"50d4c3c0b3cc602248fed09badf12150c815b13afe89b18142d276ba249cf1a2"}
May 25 13:23:53.757150 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:23:53.757106    2557 scope.go:117] "RemoveContainer" containerID="416502fb1f93226194e3ead69222ed2c9f50664e96b54d377bdc541ec5f913e8"
May 25 13:23:53.765502 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:23:53.765483    2557 scope.go:117] "RemoveContainer" containerID="416502fb1f93226194e3ead69222ed2c9f50664e96b54d377bdc541ec5f913e8"
May 25 13:23:53.765797 ip-10-0-143-235 kubenswrapper[2557]: E0525 13:23:53.765778    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"416502fb1f93226194e3ead69222ed2c9f50664e96b54d377bdc541ec5f913e8\": container with ID starting with 416502fb1f93226194e3ead69222ed2c9f50664e96b54d377bdc541ec5f913e8 not found: ID does not exist" containerID="416502fb1f93226194e3ead69222ed2c9f50664e96b54d377bdc541ec5f913e8"
May 25 13:23:53.765840 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:23:53.765806    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"416502fb1f93226194e3ead69222ed2c9f50664e96b54d377bdc541ec5f913e8"} err="failed to get container status \"416502fb1f93226194e3ead69222ed2c9f50664e96b54d377bdc541ec5f913e8\": rpc error: code = NotFound desc = could not find container \"416502fb1f93226194e3ead69222ed2c9f50664e96b54d377bdc541ec5f913e8\": container with ID starting with 416502fb1f93226194e3ead69222ed2c9f50664e96b54d377bdc541ec5f913e8 not found: ID does not exist"
May 25 13:23:53.778522 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:23:53.778496    2557 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-7cd9b78f5d-g8fjl"]
May 25 13:23:53.782651 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:23:53.782626    2557 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve/kserve-controller-manager-7cd9b78f5d-g8fjl"]
May 25 13:23:54.150628 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:23:54.150590    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e8f60531-4177-43d7-90e5-eb33298d910e" path="/var/lib/kubelet/pods/e8f60531-4177-43d7-90e5-eb33298d910e/volumes"
May 25 13:23:54.761920 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:23:54.761882    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-7cd9b78f5d-b6kcm" event={"ID":"fd65f3b4-c9f6-4141-b034-92cf6bafa8f3","Type":"ContainerStarted","Data":"b701c75a19dcabc6387d8660898fee4f2fb2f2eb92135eac54393ab2c60f1f2f"}
May 25 13:23:54.762121 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:23:54.761938    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-7cd9b78f5d-b6kcm"
May 25 13:23:54.778838 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:23:54.778785    2557 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve/kserve-controller-manager-7cd9b78f5d-b6kcm" podStartSLOduration=2.247521945 podStartE2EDuration="2.778766836s" podCreationTimestamp="2026-05-25 13:23:52 +0000 UTC" firstStartedPulling="2026-05-25 13:23:53.384918559 +0000 UTC m=+1085.847934693" lastFinishedPulling="2026-05-25 13:23:53.91616345 +0000 UTC m=+1086.379179584" observedRunningTime="2026-05-25 13:23:54.77743378 +0000 UTC m=+1087.240449936" watchObservedRunningTime="2026-05-25 13:23:54.778766836 +0000 UTC m=+1087.241782992"
May 25 13:24:25.771853 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:24:25.771822    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-7cd9b78f5d-b6kcm"
May 25 13:24:26.689135 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:24:26.689060    2557 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-86f7b4b499-kkplx"]
May 25 13:24:26.689484 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:24:26.689468    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8f60531-4177-43d7-90e5-eb33298d910e" containerName="manager"
May 25 13:24:26.689532 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:24:26.689486    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8f60531-4177-43d7-90e5-eb33298d910e" containerName="manager"
May 25 13:24:26.689609 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:24:26.689544    2557 memory_manager.go:356] "RemoveStaleState removing state" podUID="e8f60531-4177-43d7-90e5-eb33298d910e" containerName="manager"
May 25 13:24:26.692421 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:24:26.692405    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-86f7b4b499-kkplx"
May 25 13:24:26.695312 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:24:26.695286    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\""
May 25 13:24:26.695569 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:24:26.695413    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-hsght\""
May 25 13:24:26.704510 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:24:26.704485    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-86f7b4b499-kkplx"]
May 25 13:24:26.805174 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:24:26.805133    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4564t\" (UniqueName: \"kubernetes.io/projected/ee57af8c-4405-4ce5-8ed2-3c86dd820474-kube-api-access-4564t\") pod \"model-serving-api-86f7b4b499-kkplx\" (UID: \"ee57af8c-4405-4ce5-8ed2-3c86dd820474\") " pod="kserve/model-serving-api-86f7b4b499-kkplx"
May 25 13:24:26.805574 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:24:26.805246    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee57af8c-4405-4ce5-8ed2-3c86dd820474-tls-certs\") pod \"model-serving-api-86f7b4b499-kkplx\" (UID: \"ee57af8c-4405-4ce5-8ed2-3c86dd820474\") " pod="kserve/model-serving-api-86f7b4b499-kkplx"
May 25 13:24:26.906110 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:24:26.906067    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-4564t\" (UniqueName: \"kubernetes.io/projected/ee57af8c-4405-4ce5-8ed2-3c86dd820474-kube-api-access-4564t\") pod \"model-serving-api-86f7b4b499-kkplx\" (UID: \"ee57af8c-4405-4ce5-8ed2-3c86dd820474\") " pod="kserve/model-serving-api-86f7b4b499-kkplx"
May 25 13:24:26.906293 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:24:26.906155    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee57af8c-4405-4ce5-8ed2-3c86dd820474-tls-certs\") pod \"model-serving-api-86f7b4b499-kkplx\" (UID: \"ee57af8c-4405-4ce5-8ed2-3c86dd820474\") " pod="kserve/model-serving-api-86f7b4b499-kkplx"
May 25 13:24:26.906293 ip-10-0-143-235 kubenswrapper[2557]: E0525 13:24:26.906268    2557 secret.go:189] Couldn't get secret kserve/model-serving-api-tls: secret "model-serving-api-tls" not found
May 25 13:24:26.906419 ip-10-0-143-235 kubenswrapper[2557]: E0525 13:24:26.906331    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ee57af8c-4405-4ce5-8ed2-3c86dd820474-tls-certs podName:ee57af8c-4405-4ce5-8ed2-3c86dd820474 nodeName:}" failed. No retries permitted until 2026-05-25 13:24:27.406314218 +0000 UTC m=+1119.869330353 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/ee57af8c-4405-4ce5-8ed2-3c86dd820474-tls-certs") pod "model-serving-api-86f7b4b499-kkplx" (UID: "ee57af8c-4405-4ce5-8ed2-3c86dd820474") : secret "model-serving-api-tls" not found
May 25 13:24:26.915723 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:24:26.915700    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4564t\" (UniqueName: \"kubernetes.io/projected/ee57af8c-4405-4ce5-8ed2-3c86dd820474-kube-api-access-4564t\") pod \"model-serving-api-86f7b4b499-kkplx\" (UID: \"ee57af8c-4405-4ce5-8ed2-3c86dd820474\") " pod="kserve/model-serving-api-86f7b4b499-kkplx"
May 25 13:24:27.410507 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:24:27.410471    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee57af8c-4405-4ce5-8ed2-3c86dd820474-tls-certs\") pod \"model-serving-api-86f7b4b499-kkplx\" (UID: \"ee57af8c-4405-4ce5-8ed2-3c86dd820474\") " pod="kserve/model-serving-api-86f7b4b499-kkplx"
May 25 13:24:27.412866 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:24:27.412838    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee57af8c-4405-4ce5-8ed2-3c86dd820474-tls-certs\") pod \"model-serving-api-86f7b4b499-kkplx\" (UID: \"ee57af8c-4405-4ce5-8ed2-3c86dd820474\") " pod="kserve/model-serving-api-86f7b4b499-kkplx"
May 25 13:24:27.602730 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:24:27.602677    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-86f7b4b499-kkplx"
May 25 13:24:27.732960 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:24:27.732934    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-86f7b4b499-kkplx"]
May 25 13:24:27.735636 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:24:27.735609    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee57af8c_4405_4ce5_8ed2_3c86dd820474.slice/crio-78e5e6dadcbba78c431a1f1e5f9bd60d242f17df37fab0e1047c051f0c9cb828 WatchSource:0}: Error finding container 78e5e6dadcbba78c431a1f1e5f9bd60d242f17df37fab0e1047c051f0c9cb828: Status 404 returned error can't find the container with id 78e5e6dadcbba78c431a1f1e5f9bd60d242f17df37fab0e1047c051f0c9cb828
May 25 13:24:27.877329 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:24:27.877290    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-86f7b4b499-kkplx" event={"ID":"ee57af8c-4405-4ce5-8ed2-3c86dd820474","Type":"ContainerStarted","Data":"78e5e6dadcbba78c431a1f1e5f9bd60d242f17df37fab0e1047c051f0c9cb828"}
May 25 13:24:30.892407 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:24:30.892368    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-86f7b4b499-kkplx" event={"ID":"ee57af8c-4405-4ce5-8ed2-3c86dd820474","Type":"ContainerStarted","Data":"bfa64edb831451e7e1bbf978a662580c588b01a5bf7bb1712a0cb1c49997c609"}
May 25 13:24:30.892832 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:24:30.892559    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-86f7b4b499-kkplx"
May 25 13:24:30.911476 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:24:30.911418    2557 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve/model-serving-api-86f7b4b499-kkplx" podStartSLOduration=2.55119365 podStartE2EDuration="4.911404573s" podCreationTimestamp="2026-05-25 13:24:26 +0000 UTC" firstStartedPulling="2026-05-25 13:24:27.737809217 +0000 UTC m=+1120.200825366" lastFinishedPulling="2026-05-25 13:24:30.09802015 +0000 UTC m=+1122.561036289" observedRunningTime="2026-05-25 13:24:30.909035665 +0000 UTC m=+1123.372051827" watchObservedRunningTime="2026-05-25 13:24:30.911404573 +0000 UTC m=+1123.374420729"
May 25 13:24:41.900690 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:24:41.900657    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-86f7b4b499-kkplx"
May 25 13:25:08.088719 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:25:08.088635    2557 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-68966zkzwr"]
May 25 13:25:08.094667 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:25:08.094637    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-68966zkzwr"
May 25 13:25:08.097596 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:25:08.097573    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
May 25 13:25:08.097596 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:25:08.097587    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-inline-config-test-epp-sa-dockercfg-wxdq9\""
May 25 13:25:08.097887 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:25:08.097872    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
May 25 13:25:08.098649 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:25:08.098633    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-8blwv\""
May 25 13:25:08.098740 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:25:08.098726    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-inline-config-test-kserve-self-signed-certs\""
May 25 13:25:08.102965 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:25:08.102942    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-68966zkzwr"]
May 25 13:25:08.182449 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:25:08.182414    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xlbr\" (UniqueName: \"kubernetes.io/projected/b12e3a48-f3ad-49b6-916c-bef8f05408da-kube-api-access-7xlbr\") pod \"scheduler-inline-config-test-kserve-router-scheduler-68966zkzwr\" (UID: \"b12e3a48-f3ad-49b6-916c-bef8f05408da\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-68966zkzwr"
May 25 13:25:08.182647 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:25:08.182471    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/b12e3a48-f3ad-49b6-916c-bef8f05408da-tokenizer-uds\") pod \"scheduler-inline-config-test-kserve-router-scheduler-68966zkzwr\" (UID: \"b12e3a48-f3ad-49b6-916c-bef8f05408da\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-68966zkzwr"
May 25 13:25:08.182647 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:25:08.182502    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b12e3a48-f3ad-49b6-916c-bef8f05408da-tls-certs\") pod \"scheduler-inline-config-test-kserve-router-scheduler-68966zkzwr\" (UID: \"b12e3a48-f3ad-49b6-916c-bef8f05408da\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-68966zkzwr"
May 25 13:25:08.182647 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:25:08.182541    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b12e3a48-f3ad-49b6-916c-bef8f05408da-kserve-provision-location\") pod \"scheduler-inline-config-test-kserve-router-scheduler-68966zkzwr\" (UID: \"b12e3a48-f3ad-49b6-916c-bef8f05408da\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-68966zkzwr"
May 25 13:25:08.182647 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:25:08.182567    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/b12e3a48-f3ad-49b6-916c-bef8f05408da-tokenizer-tmp\") pod \"scheduler-inline-config-test-kserve-router-scheduler-68966zkzwr\" (UID: \"b12e3a48-f3ad-49b6-916c-bef8f05408da\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-68966zkzwr"
May 25 13:25:08.182647 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:25:08.182583    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/b12e3a48-f3ad-49b6-916c-bef8f05408da-tokenizer-cache\") pod \"scheduler-inline-config-test-kserve-router-scheduler-68966zkzwr\" (UID: \"b12e3a48-f3ad-49b6-916c-bef8f05408da\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-68966zkzwr"
May 25 13:25:08.283610 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:25:08.283568    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-7xlbr\" (UniqueName: \"kubernetes.io/projected/b12e3a48-f3ad-49b6-916c-bef8f05408da-kube-api-access-7xlbr\") pod \"scheduler-inline-config-test-kserve-router-scheduler-68966zkzwr\" (UID: \"b12e3a48-f3ad-49b6-916c-bef8f05408da\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-68966zkzwr"
May 25 13:25:08.283785 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:25:08.283635    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/b12e3a48-f3ad-49b6-916c-bef8f05408da-tokenizer-uds\") pod \"scheduler-inline-config-test-kserve-router-scheduler-68966zkzwr\" (UID: \"b12e3a48-f3ad-49b6-916c-bef8f05408da\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-68966zkzwr"
May 25 13:25:08.283785 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:25:08.283683    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b12e3a48-f3ad-49b6-916c-bef8f05408da-tls-certs\") pod \"scheduler-inline-config-test-kserve-router-scheduler-68966zkzwr\" (UID: \"b12e3a48-f3ad-49b6-916c-bef8f05408da\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-68966zkzwr"
May 25 13:25:08.283785 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:25:08.283742    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b12e3a48-f3ad-49b6-916c-bef8f05408da-kserve-provision-location\") pod \"scheduler-inline-config-test-kserve-router-scheduler-68966zkzwr\" (UID: \"b12e3a48-f3ad-49b6-916c-bef8f05408da\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-68966zkzwr"
May 25 13:25:08.283888 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:25:08.283787    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/b12e3a48-f3ad-49b6-916c-bef8f05408da-tokenizer-tmp\") pod \"scheduler-inline-config-test-kserve-router-scheduler-68966zkzwr\" (UID: \"b12e3a48-f3ad-49b6-916c-bef8f05408da\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-68966zkzwr"
May 25 13:25:08.283888 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:25:08.283811    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/b12e3a48-f3ad-49b6-916c-bef8f05408da-tokenizer-cache\") pod \"scheduler-inline-config-test-kserve-router-scheduler-68966zkzwr\" (UID: \"b12e3a48-f3ad-49b6-916c-bef8f05408da\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-68966zkzwr"
May 25 13:25:08.284084 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:25:08.284060    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/b12e3a48-f3ad-49b6-916c-bef8f05408da-tokenizer-uds\") pod \"scheduler-inline-config-test-kserve-router-scheduler-68966zkzwr\" (UID: \"b12e3a48-f3ad-49b6-916c-bef8f05408da\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-68966zkzwr"
May 25 13:25:08.284184 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:25:08.284162    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b12e3a48-f3ad-49b6-916c-bef8f05408da-kserve-provision-location\") pod \"scheduler-inline-config-test-kserve-router-scheduler-68966zkzwr\" (UID: \"b12e3a48-f3ad-49b6-916c-bef8f05408da\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-68966zkzwr"
May 25 13:25:08.284220 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:25:08.284179    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/b12e3a48-f3ad-49b6-916c-bef8f05408da-tokenizer-cache\") pod \"scheduler-inline-config-test-kserve-router-scheduler-68966zkzwr\" (UID: \"b12e3a48-f3ad-49b6-916c-bef8f05408da\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-68966zkzwr"
May 25 13:25:08.284220 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:25:08.284209    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/b12e3a48-f3ad-49b6-916c-bef8f05408da-tokenizer-tmp\") pod \"scheduler-inline-config-test-kserve-router-scheduler-68966zkzwr\" (UID: \"b12e3a48-f3ad-49b6-916c-bef8f05408da\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-68966zkzwr"
May 25 13:25:08.286155 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:25:08.286134    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b12e3a48-f3ad-49b6-916c-bef8f05408da-tls-certs\") pod \"scheduler-inline-config-test-kserve-router-scheduler-68966zkzwr\" (UID: \"b12e3a48-f3ad-49b6-916c-bef8f05408da\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-68966zkzwr"
May 25 13:25:08.292611 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:25:08.292577    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xlbr\" (UniqueName: \"kubernetes.io/projected/b12e3a48-f3ad-49b6-916c-bef8f05408da-kube-api-access-7xlbr\") pod \"scheduler-inline-config-test-kserve-router-scheduler-68966zkzwr\" (UID: \"b12e3a48-f3ad-49b6-916c-bef8f05408da\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-68966zkzwr"
May 25 13:25:08.406745 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:25:08.406651    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-68966zkzwr"
May 25 13:25:08.553409 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:25:08.553380    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-68966zkzwr"]
May 25 13:25:08.556108 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:25:08.556077    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb12e3a48_f3ad_49b6_916c_bef8f05408da.slice/crio-b3261f8970f72c989a1dde54c83383e5fede5003cf71df12839b14c875d5bbbd WatchSource:0}: Error finding container b3261f8970f72c989a1dde54c83383e5fede5003cf71df12839b14c875d5bbbd: Status 404 returned error can't find the container with id b3261f8970f72c989a1dde54c83383e5fede5003cf71df12839b14c875d5bbbd
May 25 13:25:09.027760 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:25:09.027722    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-68966zkzwr" event={"ID":"b12e3a48-f3ad-49b6-916c-bef8f05408da","Type":"ContainerStarted","Data":"b3261f8970f72c989a1dde54c83383e5fede5003cf71df12839b14c875d5bbbd"}
May 25 13:25:14.048253 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:25:14.048205    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-68966zkzwr" event={"ID":"b12e3a48-f3ad-49b6-916c-bef8f05408da","Type":"ContainerStarted","Data":"42cf19b2832cdfac31766a201777c41ef1976f8e451e5961c2808508918c9efe"}
May 25 13:25:15.053030 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:25:15.052993    2557 generic.go:358] "Generic (PLEG): container finished" podID="b12e3a48-f3ad-49b6-916c-bef8f05408da" containerID="42cf19b2832cdfac31766a201777c41ef1976f8e451e5961c2808508918c9efe" exitCode=0
May 25 13:25:15.053543 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:25:15.053041    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-68966zkzwr" event={"ID":"b12e3a48-f3ad-49b6-916c-bef8f05408da","Type":"ContainerDied","Data":"42cf19b2832cdfac31766a201777c41ef1976f8e451e5961c2808508918c9efe"}
May 25 13:25:15.054220 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:25:15.054204    2557 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 25 13:25:17.063884 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:25:17.063826    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-68966zkzwr" event={"ID":"b12e3a48-f3ad-49b6-916c-bef8f05408da","Type":"ContainerStarted","Data":"7b83f429398591a1c40b86a3e8b217bacb6b3a987f095b004ed41e61672413a8"}
May 25 13:25:47.186518 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:25:47.186479    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-68966zkzwr" event={"ID":"b12e3a48-f3ad-49b6-916c-bef8f05408da","Type":"ContainerStarted","Data":"bd16670b949bafe778f18586dc96e46db7ea247def1a64a5ebfeed04844ffb66"}
May 25 13:25:47.186972 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:25:47.186747    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-68966zkzwr"
May 25 13:25:47.189418 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:25:47.189390    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-68966zkzwr"
May 25 13:25:47.217433 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:25:47.217373    2557 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-68966zkzwr" podStartSLOduration=1.173259992 podStartE2EDuration="39.217357808s" podCreationTimestamp="2026-05-25 13:25:08 +0000 UTC" firstStartedPulling="2026-05-25 13:25:08.558167004 +0000 UTC m=+1161.021183160" lastFinishedPulling="2026-05-25 13:25:46.602264823 +0000 UTC m=+1199.065280976" observedRunningTime="2026-05-25 13:25:47.215301838 +0000 UTC m=+1199.678318064" watchObservedRunningTime="2026-05-25 13:25:47.217357808 +0000 UTC m=+1199.680373956"
May 25 13:25:48.120572 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:25:48.120537    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vqnvc_d21bdf8b-d0ab-48f2-8736-54e92947dd53/ovn-acl-logging/0.log"
May 25 13:25:48.122606 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:25:48.122577    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vqnvc_d21bdf8b-d0ab-48f2-8736-54e92947dd53/ovn-acl-logging/0.log"
May 25 13:25:48.407548 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:25:48.407451    2557 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-68966zkzwr"
May 25 13:25:48.407548 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:25:48.407506    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-68966zkzwr"
May 25 13:25:58.408545 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:25:58.408502    2557 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-68966zkzwr"
May 25 13:25:58.409715 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:25:58.409692    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-68966zkzwr"
May 25 13:27:09.104970 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:27:09.104931    2557 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-68966zkzwr"]
May 25 13:27:09.105415 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:27:09.105237    2557 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-68966zkzwr" podUID="b12e3a48-f3ad-49b6-916c-bef8f05408da" containerName="main" containerID="cri-o://7b83f429398591a1c40b86a3e8b217bacb6b3a987f095b004ed41e61672413a8" gracePeriod=30
May 25 13:27:09.105415 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:27:09.105279    2557 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-68966zkzwr" podUID="b12e3a48-f3ad-49b6-916c-bef8f05408da" containerName="tokenizer" containerID="cri-o://bd16670b949bafe778f18586dc96e46db7ea247def1a64a5ebfeed04844ffb66" gracePeriod=30
May 25 13:27:09.483658 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:27:09.483572    2557 generic.go:358] "Generic (PLEG): container finished" podID="b12e3a48-f3ad-49b6-916c-bef8f05408da" containerID="7b83f429398591a1c40b86a3e8b217bacb6b3a987f095b004ed41e61672413a8" exitCode=0
May 25 13:27:09.483804 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:27:09.483655    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-68966zkzwr" event={"ID":"b12e3a48-f3ad-49b6-916c-bef8f05408da","Type":"ContainerDied","Data":"7b83f429398591a1c40b86a3e8b217bacb6b3a987f095b004ed41e61672413a8"}
May 25 13:27:10.462390 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:27:10.462359    2557 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-68966zkzwr"
May 25 13:27:10.489633 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:27:10.489596    2557 generic.go:358] "Generic (PLEG): container finished" podID="b12e3a48-f3ad-49b6-916c-bef8f05408da" containerID="bd16670b949bafe778f18586dc96e46db7ea247def1a64a5ebfeed04844ffb66" exitCode=0
May 25 13:27:10.489804 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:27:10.489666    2557 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-68966zkzwr"
May 25 13:27:10.489804 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:27:10.489672    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-68966zkzwr" event={"ID":"b12e3a48-f3ad-49b6-916c-bef8f05408da","Type":"ContainerDied","Data":"bd16670b949bafe778f18586dc96e46db7ea247def1a64a5ebfeed04844ffb66"}
May 25 13:27:10.489804 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:27:10.489715    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-68966zkzwr" event={"ID":"b12e3a48-f3ad-49b6-916c-bef8f05408da","Type":"ContainerDied","Data":"b3261f8970f72c989a1dde54c83383e5fede5003cf71df12839b14c875d5bbbd"}
May 25 13:27:10.489804 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:27:10.489737    2557 scope.go:117] "RemoveContainer" containerID="bd16670b949bafe778f18586dc96e46db7ea247def1a64a5ebfeed04844ffb66"
May 25 13:27:10.498395 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:27:10.498364    2557 scope.go:117] "RemoveContainer" containerID="7b83f429398591a1c40b86a3e8b217bacb6b3a987f095b004ed41e61672413a8"
May 25 13:27:10.508864 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:27:10.508836    2557 scope.go:117] "RemoveContainer" containerID="42cf19b2832cdfac31766a201777c41ef1976f8e451e5961c2808508918c9efe"
May 25 13:27:10.516910 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:27:10.516891    2557 scope.go:117] "RemoveContainer" containerID="bd16670b949bafe778f18586dc96e46db7ea247def1a64a5ebfeed04844ffb66"
May 25 13:27:10.517234 ip-10-0-143-235 kubenswrapper[2557]: E0525 13:27:10.517212    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd16670b949bafe778f18586dc96e46db7ea247def1a64a5ebfeed04844ffb66\": container with ID starting with bd16670b949bafe778f18586dc96e46db7ea247def1a64a5ebfeed04844ffb66 not found: ID does not exist" containerID="bd16670b949bafe778f18586dc96e46db7ea247def1a64a5ebfeed04844ffb66"
May 25 13:27:10.517328 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:27:10.517239    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd16670b949bafe778f18586dc96e46db7ea247def1a64a5ebfeed04844ffb66"} err="failed to get container status \"bd16670b949bafe778f18586dc96e46db7ea247def1a64a5ebfeed04844ffb66\": rpc error: code = NotFound desc = could not find container \"bd16670b949bafe778f18586dc96e46db7ea247def1a64a5ebfeed04844ffb66\": container with ID starting with bd16670b949bafe778f18586dc96e46db7ea247def1a64a5ebfeed04844ffb66 not found: ID does not exist"
May 25 13:27:10.517328 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:27:10.517261    2557 scope.go:117] "RemoveContainer" containerID="7b83f429398591a1c40b86a3e8b217bacb6b3a987f095b004ed41e61672413a8"
May 25 13:27:10.517519 ip-10-0-143-235 kubenswrapper[2557]: E0525 13:27:10.517499    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b83f429398591a1c40b86a3e8b217bacb6b3a987f095b004ed41e61672413a8\": container with ID starting with 7b83f429398591a1c40b86a3e8b217bacb6b3a987f095b004ed41e61672413a8 not found: ID does not exist" containerID="7b83f429398591a1c40b86a3e8b217bacb6b3a987f095b004ed41e61672413a8"
May 25 13:27:10.517568 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:27:10.517526    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b83f429398591a1c40b86a3e8b217bacb6b3a987f095b004ed41e61672413a8"} err="failed to get container status \"7b83f429398591a1c40b86a3e8b217bacb6b3a987f095b004ed41e61672413a8\": rpc error: code = NotFound desc = could not find container \"7b83f429398591a1c40b86a3e8b217bacb6b3a987f095b004ed41e61672413a8\": container with ID starting with 7b83f429398591a1c40b86a3e8b217bacb6b3a987f095b004ed41e61672413a8 not found: ID does not exist"
May 25 13:27:10.517568 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:27:10.517540    2557 scope.go:117] "RemoveContainer" containerID="42cf19b2832cdfac31766a201777c41ef1976f8e451e5961c2808508918c9efe"
May 25 13:27:10.517774 ip-10-0-143-235 kubenswrapper[2557]: E0525 13:27:10.517757    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42cf19b2832cdfac31766a201777c41ef1976f8e451e5961c2808508918c9efe\": container with ID starting with 42cf19b2832cdfac31766a201777c41ef1976f8e451e5961c2808508918c9efe not found: ID does not exist" containerID="42cf19b2832cdfac31766a201777c41ef1976f8e451e5961c2808508918c9efe"
May 25 13:27:10.517811 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:27:10.517778    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42cf19b2832cdfac31766a201777c41ef1976f8e451e5961c2808508918c9efe"} err="failed to get container status \"42cf19b2832cdfac31766a201777c41ef1976f8e451e5961c2808508918c9efe\": rpc error: code = NotFound desc = could not find container \"42cf19b2832cdfac31766a201777c41ef1976f8e451e5961c2808508918c9efe\": container with ID starting with 42cf19b2832cdfac31766a201777c41ef1976f8e451e5961c2808508918c9efe not found: ID does not exist"
May 25 13:27:10.537207 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:27:10.537121    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b12e3a48-f3ad-49b6-916c-bef8f05408da-tls-certs\") pod \"b12e3a48-f3ad-49b6-916c-bef8f05408da\" (UID: \"b12e3a48-f3ad-49b6-916c-bef8f05408da\") "
May 25 13:27:10.537207 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:27:10.537174    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/b12e3a48-f3ad-49b6-916c-bef8f05408da-tokenizer-cache\") pod \"b12e3a48-f3ad-49b6-916c-bef8f05408da\" (UID: \"b12e3a48-f3ad-49b6-916c-bef8f05408da\") "
May 25 13:27:10.537436 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:27:10.537218    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/b12e3a48-f3ad-49b6-916c-bef8f05408da-tokenizer-uds\") pod \"b12e3a48-f3ad-49b6-916c-bef8f05408da\" (UID: \"b12e3a48-f3ad-49b6-916c-bef8f05408da\") "
May 25 13:27:10.537436 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:27:10.537237    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/b12e3a48-f3ad-49b6-916c-bef8f05408da-tokenizer-tmp\") pod \"b12e3a48-f3ad-49b6-916c-bef8f05408da\" (UID: \"b12e3a48-f3ad-49b6-916c-bef8f05408da\") "
May 25 13:27:10.537436 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:27:10.537256    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b12e3a48-f3ad-49b6-916c-bef8f05408da-kserve-provision-location\") pod \"b12e3a48-f3ad-49b6-916c-bef8f05408da\" (UID: \"b12e3a48-f3ad-49b6-916c-bef8f05408da\") "
May 25 13:27:10.537436 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:27:10.537295    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7xlbr\" (UniqueName: \"kubernetes.io/projected/b12e3a48-f3ad-49b6-916c-bef8f05408da-kube-api-access-7xlbr\") pod \"b12e3a48-f3ad-49b6-916c-bef8f05408da\" (UID: \"b12e3a48-f3ad-49b6-916c-bef8f05408da\") "
May 25 13:27:10.537646 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:27:10.537582    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b12e3a48-f3ad-49b6-916c-bef8f05408da-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "b12e3a48-f3ad-49b6-916c-bef8f05408da" (UID: "b12e3a48-f3ad-49b6-916c-bef8f05408da"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 13:27:10.537646 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:27:10.537604    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b12e3a48-f3ad-49b6-916c-bef8f05408da-tokenizer-cache" (OuterVolumeSpecName: "tokenizer-cache") pod "b12e3a48-f3ad-49b6-916c-bef8f05408da" (UID: "b12e3a48-f3ad-49b6-916c-bef8f05408da"). InnerVolumeSpecName "tokenizer-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 13:27:10.537756 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:27:10.537691    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b12e3a48-f3ad-49b6-916c-bef8f05408da-tokenizer-tmp" (OuterVolumeSpecName: "tokenizer-tmp") pod "b12e3a48-f3ad-49b6-916c-bef8f05408da" (UID: "b12e3a48-f3ad-49b6-916c-bef8f05408da"). InnerVolumeSpecName "tokenizer-tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 13:27:10.537756 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:27:10.537699    2557 reconciler_common.go:299] "Volume detached for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/b12e3a48-f3ad-49b6-916c-bef8f05408da-tokenizer-cache\") on node \"ip-10-0-143-235.ec2.internal\" DevicePath \"\""
May 25 13:27:10.537756 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:27:10.537718    2557 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/b12e3a48-f3ad-49b6-916c-bef8f05408da-tokenizer-uds\") on node \"ip-10-0-143-235.ec2.internal\" DevicePath \"\""
May 25 13:27:10.538192 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:27:10.538165    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b12e3a48-f3ad-49b6-916c-bef8f05408da-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b12e3a48-f3ad-49b6-916c-bef8f05408da" (UID: "b12e3a48-f3ad-49b6-916c-bef8f05408da"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 13:27:10.539590 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:27:10.539568    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b12e3a48-f3ad-49b6-916c-bef8f05408da-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b12e3a48-f3ad-49b6-916c-bef8f05408da" (UID: "b12e3a48-f3ad-49b6-916c-bef8f05408da"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 13:27:10.539684 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:27:10.539606    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b12e3a48-f3ad-49b6-916c-bef8f05408da-kube-api-access-7xlbr" (OuterVolumeSpecName: "kube-api-access-7xlbr") pod "b12e3a48-f3ad-49b6-916c-bef8f05408da" (UID: "b12e3a48-f3ad-49b6-916c-bef8f05408da"). InnerVolumeSpecName "kube-api-access-7xlbr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 13:27:10.638757 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:27:10.638718    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b12e3a48-f3ad-49b6-916c-bef8f05408da-tls-certs\") on node \"ip-10-0-143-235.ec2.internal\" DevicePath \"\""
May 25 13:27:10.638757 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:27:10.638751    2557 reconciler_common.go:299] "Volume detached for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/b12e3a48-f3ad-49b6-916c-bef8f05408da-tokenizer-tmp\") on node \"ip-10-0-143-235.ec2.internal\" DevicePath \"\""
May 25 13:27:10.638757 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:27:10.638761    2557 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b12e3a48-f3ad-49b6-916c-bef8f05408da-kserve-provision-location\") on node \"ip-10-0-143-235.ec2.internal\" DevicePath \"\""
May 25 13:27:10.638992 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:27:10.638772    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7xlbr\" (UniqueName: \"kubernetes.io/projected/b12e3a48-f3ad-49b6-916c-bef8f05408da-kube-api-access-7xlbr\") on node \"ip-10-0-143-235.ec2.internal\" DevicePath \"\""
May 25 13:27:10.813498 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:27:10.813462    2557 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-68966zkzwr"]
May 25 13:27:10.817505 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:27:10.817473    2557 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-68966zkzwr"]
May 25 13:27:12.150834 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:27:12.150785    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b12e3a48-f3ad-49b6-916c-bef8f05408da" path="/var/lib/kubelet/pods/b12e3a48-f3ad-49b6-916c-bef8f05408da/volumes"
May 25 13:27:17.973778 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:27:17.973734    2557 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-7db8dcsfcx"]
May 25 13:27:17.974395 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:27:17.974373    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b12e3a48-f3ad-49b6-916c-bef8f05408da" containerName="main"
May 25 13:27:17.974446 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:27:17.974397    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="b12e3a48-f3ad-49b6-916c-bef8f05408da" containerName="main"
May 25 13:27:17.974446 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:27:17.974419    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b12e3a48-f3ad-49b6-916c-bef8f05408da" containerName="storage-initializer"
May 25 13:27:17.974446 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:27:17.974429    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="b12e3a48-f3ad-49b6-916c-bef8f05408da" containerName="storage-initializer"
May 25 13:27:17.974446 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:27:17.974441    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b12e3a48-f3ad-49b6-916c-bef8f05408da" containerName="tokenizer"
May 25 13:27:17.974567 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:27:17.974450    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="b12e3a48-f3ad-49b6-916c-bef8f05408da" containerName="tokenizer"
May 25 13:27:17.974599 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:27:17.974572    2557 memory_manager.go:356] "RemoveStaleState removing state" podUID="b12e3a48-f3ad-49b6-916c-bef8f05408da" containerName="tokenizer"
May 25 13:27:17.974599 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:27:17.974591    2557 memory_manager.go:356] "RemoveStaleState removing state" podUID="b12e3a48-f3ad-49b6-916c-bef8f05408da" containerName="main"
May 25 13:27:17.977571 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:27:17.977540    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-7db8dcsfcx"
May 25 13:27:17.980228 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:27:17.980205    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
May 25 13:27:17.981432 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:27:17.981410    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-configmap-ref-test-epp-sa-dockercfg-8cgrq\""
May 25 13:27:17.981544 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:27:17.981433    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
May 25 13:27:17.981544 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:27:17.981437    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-configmap-ref-test-kserve-self-signed-certs\""
May 25 13:27:17.981544 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:27:17.981416    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-8blwv\""
May 25 13:27:17.988826 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:27:17.988802    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-7db8dcsfcx"]
May 25 13:27:18.108547 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:27:18.108501    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cacceccb-1084-4014-827d-e6ce1e92f795-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-7db8dcsfcx\" (UID: \"cacceccb-1084-4014-827d-e6ce1e92f795\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-7db8dcsfcx"
May 25 13:27:18.108547 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:27:18.108546    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zb9sh\" (UniqueName: \"kubernetes.io/projected/cacceccb-1084-4014-827d-e6ce1e92f795-kube-api-access-zb9sh\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-7db8dcsfcx\" (UID: \"cacceccb-1084-4014-827d-e6ce1e92f795\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-7db8dcsfcx"
May 25 13:27:18.108757 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:27:18.108572    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/cacceccb-1084-4014-827d-e6ce1e92f795-tokenizer-cache\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-7db8dcsfcx\" (UID: \"cacceccb-1084-4014-827d-e6ce1e92f795\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-7db8dcsfcx"
May 25 13:27:18.108757 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:27:18.108646    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/cacceccb-1084-4014-827d-e6ce1e92f795-tokenizer-uds\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-7db8dcsfcx\" (UID: \"cacceccb-1084-4014-827d-e6ce1e92f795\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-7db8dcsfcx"
May 25 13:27:18.108757 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:27:18.108706    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/cacceccb-1084-4014-827d-e6ce1e92f795-tokenizer-tmp\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-7db8dcsfcx\" (UID: \"cacceccb-1084-4014-827d-e6ce1e92f795\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-7db8dcsfcx"
May 25 13:27:18.108855 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:27:18.108758    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cacceccb-1084-4014-827d-e6ce1e92f795-kserve-provision-location\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-7db8dcsfcx\" (UID: \"cacceccb-1084-4014-827d-e6ce1e92f795\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-7db8dcsfcx"
May 25 13:27:18.209775 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:27:18.209733    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cacceccb-1084-4014-827d-e6ce1e92f795-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-7db8dcsfcx\" (UID: \"cacceccb-1084-4014-827d-e6ce1e92f795\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-7db8dcsfcx"
May 25 13:27:18.209970 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:27:18.209783    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-zb9sh\" (UniqueName: \"kubernetes.io/projected/cacceccb-1084-4014-827d-e6ce1e92f795-kube-api-access-zb9sh\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-7db8dcsfcx\" (UID: \"cacceccb-1084-4014-827d-e6ce1e92f795\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-7db8dcsfcx"
May 25 13:27:18.209970 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:27:18.209825    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/cacceccb-1084-4014-827d-e6ce1e92f795-tokenizer-cache\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-7db8dcsfcx\" (UID: \"cacceccb-1084-4014-827d-e6ce1e92f795\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-7db8dcsfcx"
May 25 13:27:18.209970 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:27:18.209937    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/cacceccb-1084-4014-827d-e6ce1e92f795-tokenizer-uds\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-7db8dcsfcx\" (UID: \"cacceccb-1084-4014-827d-e6ce1e92f795\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-7db8dcsfcx"
May 25 13:27:18.210163 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:27:18.210004    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/cacceccb-1084-4014-827d-e6ce1e92f795-tokenizer-tmp\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-7db8dcsfcx\" (UID: \"cacceccb-1084-4014-827d-e6ce1e92f795\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-7db8dcsfcx"
May 25 13:27:18.210163 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:27:18.210048    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cacceccb-1084-4014-827d-e6ce1e92f795-kserve-provision-location\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-7db8dcsfcx\" (UID: \"cacceccb-1084-4014-827d-e6ce1e92f795\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-7db8dcsfcx"
May 25 13:27:18.210551 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:27:18.210275    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/cacceccb-1084-4014-827d-e6ce1e92f795-tokenizer-cache\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-7db8dcsfcx\" (UID: \"cacceccb-1084-4014-827d-e6ce1e92f795\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-7db8dcsfcx"
May 25 13:27:18.210551 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:27:18.210497    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cacceccb-1084-4014-827d-e6ce1e92f795-kserve-provision-location\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-7db8dcsfcx\" (UID: \"cacceccb-1084-4014-827d-e6ce1e92f795\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-7db8dcsfcx"
May 25 13:27:18.210691 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:27:18.210598    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/cacceccb-1084-4014-827d-e6ce1e92f795-tokenizer-uds\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-7db8dcsfcx\" (UID: \"cacceccb-1084-4014-827d-e6ce1e92f795\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-7db8dcsfcx"
May 25 13:27:18.210691 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:27:18.210624    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/cacceccb-1084-4014-827d-e6ce1e92f795-tokenizer-tmp\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-7db8dcsfcx\" (UID: \"cacceccb-1084-4014-827d-e6ce1e92f795\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-7db8dcsfcx"
May 25 13:27:18.212281 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:27:18.212261    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cacceccb-1084-4014-827d-e6ce1e92f795-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-7db8dcsfcx\" (UID: \"cacceccb-1084-4014-827d-e6ce1e92f795\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-7db8dcsfcx"
May 25 13:27:18.217839 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:27:18.217815    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zb9sh\" (UniqueName: \"kubernetes.io/projected/cacceccb-1084-4014-827d-e6ce1e92f795-kube-api-access-zb9sh\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-7db8dcsfcx\" (UID: \"cacceccb-1084-4014-827d-e6ce1e92f795\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-7db8dcsfcx"
May 25 13:27:18.289542 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:27:18.289449    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-7db8dcsfcx"
May 25 13:27:18.428738 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:27:18.428712    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-7db8dcsfcx"]
May 25 13:27:18.431110 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:27:18.431083    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcacceccb_1084_4014_827d_e6ce1e92f795.slice/crio-865d4c115816abb6ffd8bddd95784c434d3ebc8cd1c47358c01de0c5fe9ee9fc WatchSource:0}: Error finding container 865d4c115816abb6ffd8bddd95784c434d3ebc8cd1c47358c01de0c5fe9ee9fc: Status 404 returned error can't find the container with id 865d4c115816abb6ffd8bddd95784c434d3ebc8cd1c47358c01de0c5fe9ee9fc
May 25 13:27:18.522800 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:27:18.522754    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-7db8dcsfcx" event={"ID":"cacceccb-1084-4014-827d-e6ce1e92f795","Type":"ContainerStarted","Data":"6b7c8c0de4f2b8d099a997c3f29ac3f757cb2eadffc9019365d8117aeb8ff241"}
May 25 13:27:18.522964 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:27:18.522806    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-7db8dcsfcx" event={"ID":"cacceccb-1084-4014-827d-e6ce1e92f795","Type":"ContainerStarted","Data":"865d4c115816abb6ffd8bddd95784c434d3ebc8cd1c47358c01de0c5fe9ee9fc"}
May 25 13:27:19.527489 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:27:19.527410    2557 generic.go:358] "Generic (PLEG): container finished" podID="cacceccb-1084-4014-827d-e6ce1e92f795" containerID="6b7c8c0de4f2b8d099a997c3f29ac3f757cb2eadffc9019365d8117aeb8ff241" exitCode=0
May 25 13:27:19.527851 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:27:19.527485    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-7db8dcsfcx" event={"ID":"cacceccb-1084-4014-827d-e6ce1e92f795","Type":"ContainerDied","Data":"6b7c8c0de4f2b8d099a997c3f29ac3f757cb2eadffc9019365d8117aeb8ff241"}
May 25 13:27:20.532582 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:27:20.532542    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-7db8dcsfcx" event={"ID":"cacceccb-1084-4014-827d-e6ce1e92f795","Type":"ContainerStarted","Data":"028abd1df64aaed5d32a63d5e34ad67ec195b9fc365e2224ea750477a06fb79f"}
May 25 13:27:20.532582 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:27:20.532581    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-7db8dcsfcx" event={"ID":"cacceccb-1084-4014-827d-e6ce1e92f795","Type":"ContainerStarted","Data":"93d67d53ec073097a9722982e7db613088317a1d3e2b1f29fa9e41629477c114"}
May 25 13:27:20.533088 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:27:20.532621    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-7db8dcsfcx"
May 25 13:27:20.555562 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:27:20.555496    2557 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-7db8dcsfcx" podStartSLOduration=3.5554731889999998 podStartE2EDuration="3.555473189s" podCreationTimestamp="2026-05-25 13:27:17 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 13:27:20.552945442 +0000 UTC m=+1293.015961636" watchObservedRunningTime="2026-05-25 13:27:20.555473189 +0000 UTC m=+1293.018489348"
May 25 13:27:28.290401 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:27:28.290333    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-7db8dcsfcx"
May 25 13:27:28.290845 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:27:28.290495    2557 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-7db8dcsfcx"
May 25 13:27:28.293202 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:27:28.293178    2557 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-7db8dcsfcx"
May 25 13:27:28.563241 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:27:28.563205    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-7db8dcsfcx"
May 25 13:27:49.970858 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:27:49.970823    2557 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schexf6gg"]
May 25 13:27:49.974191 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:27:49.974164    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schexf6gg"
May 25 13:27:49.977537 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:27:49.977515    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-fb-opt-125m-route-55f7ae4a-epp-sa-dockercfg-d7wkl\""
May 25 13:27:49.978697 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:27:49.978680    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvdde380eaa9fe1facad32d45131f9e34d-kserve-self-signed-certs\""
May 25 13:27:49.985507 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:27:49.985481    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schexf6gg"]
May 25 13:27:49.994983 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:27:49.994947    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6b7779d-d70f-4a73-960e-9b1e13044b28-tls-certs\") pod \"llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schexf6gg\" (UID: \"e6b7779d-d70f-4a73-960e-9b1e13044b28\") " pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schexf6gg"
May 25 13:27:49.995149 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:27:49.995056    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tgl8n\" (UniqueName: \"kubernetes.io/projected/e6b7779d-d70f-4a73-960e-9b1e13044b28-kube-api-access-tgl8n\") pod \"llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schexf6gg\" (UID: \"e6b7779d-d70f-4a73-960e-9b1e13044b28\") " pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schexf6gg"
May 25 13:27:49.995149 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:27:49.995113    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/e6b7779d-d70f-4a73-960e-9b1e13044b28-tokenizer-tmp\") pod \"llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schexf6gg\" (UID: \"e6b7779d-d70f-4a73-960e-9b1e13044b28\") " pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schexf6gg"
May 25 13:27:49.995239 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:27:49.995146    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6b7779d-d70f-4a73-960e-9b1e13044b28-tokenizer-cache\") pod \"llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schexf6gg\" (UID: \"e6b7779d-d70f-4a73-960e-9b1e13044b28\") " pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schexf6gg"
May 25 13:27:49.995239 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:27:49.995195    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e6b7779d-d70f-4a73-960e-9b1e13044b28-kserve-provision-location\") pod \"llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schexf6gg\" (UID: \"e6b7779d-d70f-4a73-960e-9b1e13044b28\") " pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schexf6gg"
May 25 13:27:49.995313 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:27:49.995251    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/e6b7779d-d70f-4a73-960e-9b1e13044b28-tokenizer-uds\") pod \"llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schexf6gg\" (UID: \"e6b7779d-d70f-4a73-960e-9b1e13044b28\") " pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schexf6gg"
May 25 13:27:50.096073 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:27:50.096035    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-tgl8n\" (UniqueName: \"kubernetes.io/projected/e6b7779d-d70f-4a73-960e-9b1e13044b28-kube-api-access-tgl8n\") pod \"llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schexf6gg\" (UID: \"e6b7779d-d70f-4a73-960e-9b1e13044b28\") " pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schexf6gg"
May 25 13:27:50.096297 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:27:50.096209    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/e6b7779d-d70f-4a73-960e-9b1e13044b28-tokenizer-tmp\") pod \"llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schexf6gg\" (UID: \"e6b7779d-d70f-4a73-960e-9b1e13044b28\") " pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schexf6gg"
May 25 13:27:50.096297 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:27:50.096257    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6b7779d-d70f-4a73-960e-9b1e13044b28-tokenizer-cache\") pod \"llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schexf6gg\" (UID: \"e6b7779d-d70f-4a73-960e-9b1e13044b28\") " pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schexf6gg"
May 25 13:27:50.096451 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:27:50.096314    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e6b7779d-d70f-4a73-960e-9b1e13044b28-kserve-provision-location\") pod \"llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schexf6gg\" (UID: \"e6b7779d-d70f-4a73-960e-9b1e13044b28\") " pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schexf6gg"
May 25 13:27:50.096451 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:27:50.096437    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/e6b7779d-d70f-4a73-960e-9b1e13044b28-tokenizer-uds\") pod \"llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schexf6gg\" (UID: \"e6b7779d-d70f-4a73-960e-9b1e13044b28\") " pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schexf6gg"
May 25 13:27:50.096584 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:27:50.096561    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6b7779d-d70f-4a73-960e-9b1e13044b28-tls-certs\") pod \"llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schexf6gg\" (UID: \"e6b7779d-d70f-4a73-960e-9b1e13044b28\") " pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schexf6gg"
May 25 13:27:50.096706 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:27:50.096683    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/e6b7779d-d70f-4a73-960e-9b1e13044b28-tokenizer-tmp\") pod \"llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schexf6gg\" (UID: \"e6b7779d-d70f-4a73-960e-9b1e13044b28\") " pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schexf6gg"
May 25 13:27:50.096881 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:27:50.096851    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/e6b7779d-d70f-4a73-960e-9b1e13044b28-tokenizer-uds\") pod \"llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schexf6gg\" (UID: \"e6b7779d-d70f-4a73-960e-9b1e13044b28\") " pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schexf6gg"
May 25 13:27:50.096975 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:27:50.096754    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6b7779d-d70f-4a73-960e-9b1e13044b28-tokenizer-cache\") pod \"llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schexf6gg\" (UID: \"e6b7779d-d70f-4a73-960e-9b1e13044b28\") " pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schexf6gg"
May 25 13:27:50.096975 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:27:50.096908    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e6b7779d-d70f-4a73-960e-9b1e13044b28-kserve-provision-location\") pod \"llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schexf6gg\" (UID: \"e6b7779d-d70f-4a73-960e-9b1e13044b28\") " pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schexf6gg"
May 25 13:27:50.099039 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:27:50.099018    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6b7779d-d70f-4a73-960e-9b1e13044b28-tls-certs\") pod \"llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schexf6gg\" (UID: \"e6b7779d-d70f-4a73-960e-9b1e13044b28\") " pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schexf6gg"
May 25 13:27:50.108310 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:27:50.108284    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgl8n\" (UniqueName: \"kubernetes.io/projected/e6b7779d-d70f-4a73-960e-9b1e13044b28-kube-api-access-tgl8n\") pod \"llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schexf6gg\" (UID: \"e6b7779d-d70f-4a73-960e-9b1e13044b28\") " pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schexf6gg"
May 25 13:27:50.284609 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:27:50.284513    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schexf6gg"
May 25 13:27:50.425868 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:27:50.425838    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schexf6gg"]
May 25 13:27:50.427646 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:27:50.427618    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6b7779d_d70f_4a73_960e_9b1e13044b28.slice/crio-68b40df2cefa53accf5e1d5b24edeeb98c2b02eaf7a8a0104cbe850d89b521e7 WatchSource:0}: Error finding container 68b40df2cefa53accf5e1d5b24edeeb98c2b02eaf7a8a0104cbe850d89b521e7: Status 404 returned error can't find the container with id 68b40df2cefa53accf5e1d5b24edeeb98c2b02eaf7a8a0104cbe850d89b521e7
May 25 13:27:50.570997 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:27:50.570966    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-7db8dcsfcx"
May 25 13:27:50.646003 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:27:50.645968    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schexf6gg" event={"ID":"e6b7779d-d70f-4a73-960e-9b1e13044b28","Type":"ContainerStarted","Data":"81057c4d50a1166664e47323146c5e63f19f800b13796a58e8dda4db08c1f271"}
May 25 13:27:50.646003 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:27:50.646011    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schexf6gg" event={"ID":"e6b7779d-d70f-4a73-960e-9b1e13044b28","Type":"ContainerStarted","Data":"68b40df2cefa53accf5e1d5b24edeeb98c2b02eaf7a8a0104cbe850d89b521e7"}
May 25 13:27:51.651030 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:27:51.650996    2557 generic.go:358] "Generic (PLEG): container finished" podID="e6b7779d-d70f-4a73-960e-9b1e13044b28" containerID="81057c4d50a1166664e47323146c5e63f19f800b13796a58e8dda4db08c1f271" exitCode=0
May 25 13:27:51.651442 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:27:51.651084    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schexf6gg" event={"ID":"e6b7779d-d70f-4a73-960e-9b1e13044b28","Type":"ContainerDied","Data":"81057c4d50a1166664e47323146c5e63f19f800b13796a58e8dda4db08c1f271"}
May 25 13:27:52.657098 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:27:52.657062    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schexf6gg" event={"ID":"e6b7779d-d70f-4a73-960e-9b1e13044b28","Type":"ContainerStarted","Data":"3c302ef84d4cd42709f164d26a7090aab9ced5539eb30d868ae91555e0463355"}
May 25 13:27:52.657098 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:27:52.657106    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schexf6gg" event={"ID":"e6b7779d-d70f-4a73-960e-9b1e13044b28","Type":"ContainerStarted","Data":"5241c281d3792568747e2f92664d979fbe725f0c426513177a459f03d05ef10b"}
May 25 13:27:52.657658 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:27:52.657234    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schexf6gg"
May 25 13:27:52.681648 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:27:52.681589    2557 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schexf6gg" podStartSLOduration=3.681573657 podStartE2EDuration="3.681573657s" podCreationTimestamp="2026-05-25 13:27:49 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 13:27:52.679173243 +0000 UTC m=+1325.142189437" watchObservedRunningTime="2026-05-25 13:27:52.681573657 +0000 UTC m=+1325.144589869"
May 25 13:28:00.285062 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:28:00.285017    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schexf6gg"
May 25 13:28:00.285062 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:28:00.285073    2557 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schexf6gg"
May 25 13:28:00.287966 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:28:00.287941    2557 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schexf6gg"
May 25 13:28:00.691462 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:28:00.691426    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schexf6gg"
May 25 13:28:01.771536 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:28:01.771498    2557 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-7db8dcsfcx"]
May 25 13:28:01.772015 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:28:01.771963    2557 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-7db8dcsfcx" podUID="cacceccb-1084-4014-827d-e6ce1e92f795" containerName="main" containerID="cri-o://93d67d53ec073097a9722982e7db613088317a1d3e2b1f29fa9e41629477c114" gracePeriod=30
May 25 13:28:01.772115 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:28:01.771985    2557 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-7db8dcsfcx" podUID="cacceccb-1084-4014-827d-e6ce1e92f795" containerName="tokenizer" containerID="cri-o://028abd1df64aaed5d32a63d5e34ad67ec195b9fc365e2224ea750477a06fb79f" gracePeriod=30
May 25 13:28:02.700191 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:28:02.700158    2557 generic.go:358] "Generic (PLEG): container finished" podID="cacceccb-1084-4014-827d-e6ce1e92f795" containerID="93d67d53ec073097a9722982e7db613088317a1d3e2b1f29fa9e41629477c114" exitCode=0
May 25 13:28:02.700379 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:28:02.700242    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-7db8dcsfcx" event={"ID":"cacceccb-1084-4014-827d-e6ce1e92f795","Type":"ContainerDied","Data":"93d67d53ec073097a9722982e7db613088317a1d3e2b1f29fa9e41629477c114"}
May 25 13:28:03.121239 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:28:03.121215    2557 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-7db8dcsfcx"
May 25 13:28:03.223562 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:28:03.223526    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zb9sh\" (UniqueName: \"kubernetes.io/projected/cacceccb-1084-4014-827d-e6ce1e92f795-kube-api-access-zb9sh\") pod \"cacceccb-1084-4014-827d-e6ce1e92f795\" (UID: \"cacceccb-1084-4014-827d-e6ce1e92f795\") "
May 25 13:28:03.223747 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:28:03.223570    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cacceccb-1084-4014-827d-e6ce1e92f795-kserve-provision-location\") pod \"cacceccb-1084-4014-827d-e6ce1e92f795\" (UID: \"cacceccb-1084-4014-827d-e6ce1e92f795\") "
May 25 13:28:03.223747 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:28:03.223639    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cacceccb-1084-4014-827d-e6ce1e92f795-tls-certs\") pod \"cacceccb-1084-4014-827d-e6ce1e92f795\" (UID: \"cacceccb-1084-4014-827d-e6ce1e92f795\") "
May 25 13:28:03.223747 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:28:03.223665    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/cacceccb-1084-4014-827d-e6ce1e92f795-tokenizer-uds\") pod \"cacceccb-1084-4014-827d-e6ce1e92f795\" (UID: \"cacceccb-1084-4014-827d-e6ce1e92f795\") "
May 25 13:28:03.223747 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:28:03.223700    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/cacceccb-1084-4014-827d-e6ce1e92f795-tokenizer-tmp\") pod \"cacceccb-1084-4014-827d-e6ce1e92f795\" (UID: \"cacceccb-1084-4014-827d-e6ce1e92f795\") "
May 25 13:28:03.223747 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:28:03.223728    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/cacceccb-1084-4014-827d-e6ce1e92f795-tokenizer-cache\") pod \"cacceccb-1084-4014-827d-e6ce1e92f795\" (UID: \"cacceccb-1084-4014-827d-e6ce1e92f795\") "
May 25 13:28:03.224021 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:28:03.223868    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cacceccb-1084-4014-827d-e6ce1e92f795-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "cacceccb-1084-4014-827d-e6ce1e92f795" (UID: "cacceccb-1084-4014-827d-e6ce1e92f795"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 13:28:03.224091 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:28:03.224048    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cacceccb-1084-4014-827d-e6ce1e92f795-tokenizer-tmp" (OuterVolumeSpecName: "tokenizer-tmp") pod "cacceccb-1084-4014-827d-e6ce1e92f795" (UID: "cacceccb-1084-4014-827d-e6ce1e92f795"). InnerVolumeSpecName "tokenizer-tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 13:28:03.224172 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:28:03.224131    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cacceccb-1084-4014-827d-e6ce1e92f795-tokenizer-cache" (OuterVolumeSpecName: "tokenizer-cache") pod "cacceccb-1084-4014-827d-e6ce1e92f795" (UID: "cacceccb-1084-4014-827d-e6ce1e92f795"). InnerVolumeSpecName "tokenizer-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 13:28:03.224335 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:28:03.224314    2557 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/cacceccb-1084-4014-827d-e6ce1e92f795-tokenizer-uds\") on node \"ip-10-0-143-235.ec2.internal\" DevicePath \"\""
May 25 13:28:03.224522 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:28:03.224501    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cacceccb-1084-4014-827d-e6ce1e92f795-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "cacceccb-1084-4014-827d-e6ce1e92f795" (UID: "cacceccb-1084-4014-827d-e6ce1e92f795"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 13:28:03.225907 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:28:03.225876    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cacceccb-1084-4014-827d-e6ce1e92f795-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "cacceccb-1084-4014-827d-e6ce1e92f795" (UID: "cacceccb-1084-4014-827d-e6ce1e92f795"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 13:28:03.226014 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:28:03.225909    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cacceccb-1084-4014-827d-e6ce1e92f795-kube-api-access-zb9sh" (OuterVolumeSpecName: "kube-api-access-zb9sh") pod "cacceccb-1084-4014-827d-e6ce1e92f795" (UID: "cacceccb-1084-4014-827d-e6ce1e92f795"). InnerVolumeSpecName "kube-api-access-zb9sh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 13:28:03.324951 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:28:03.324916    2557 reconciler_common.go:299] "Volume detached for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/cacceccb-1084-4014-827d-e6ce1e92f795-tokenizer-tmp\") on node \"ip-10-0-143-235.ec2.internal\" DevicePath \"\""
May 25 13:28:03.324951 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:28:03.324948    2557 reconciler_common.go:299] "Volume detached for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/cacceccb-1084-4014-827d-e6ce1e92f795-tokenizer-cache\") on node \"ip-10-0-143-235.ec2.internal\" DevicePath \"\""
May 25 13:28:03.325146 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:28:03.324959    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zb9sh\" (UniqueName: \"kubernetes.io/projected/cacceccb-1084-4014-827d-e6ce1e92f795-kube-api-access-zb9sh\") on node \"ip-10-0-143-235.ec2.internal\" DevicePath \"\""
May 25 13:28:03.325146 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:28:03.324970    2557 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cacceccb-1084-4014-827d-e6ce1e92f795-kserve-provision-location\") on node \"ip-10-0-143-235.ec2.internal\" DevicePath \"\""
May 25 13:28:03.325146 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:28:03.324980    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cacceccb-1084-4014-827d-e6ce1e92f795-tls-certs\") on node \"ip-10-0-143-235.ec2.internal\" DevicePath \"\""
May 25 13:28:03.705535 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:28:03.705449    2557 generic.go:358] "Generic (PLEG): container finished" podID="cacceccb-1084-4014-827d-e6ce1e92f795" containerID="028abd1df64aaed5d32a63d5e34ad67ec195b9fc365e2224ea750477a06fb79f" exitCode=0
May 25 13:28:03.705682 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:28:03.705535    2557 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-7db8dcsfcx"
May 25 13:28:03.705682 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:28:03.705534    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-7db8dcsfcx" event={"ID":"cacceccb-1084-4014-827d-e6ce1e92f795","Type":"ContainerDied","Data":"028abd1df64aaed5d32a63d5e34ad67ec195b9fc365e2224ea750477a06fb79f"}
May 25 13:28:03.705682 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:28:03.705654    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-7db8dcsfcx" event={"ID":"cacceccb-1084-4014-827d-e6ce1e92f795","Type":"ContainerDied","Data":"865d4c115816abb6ffd8bddd95784c434d3ebc8cd1c47358c01de0c5fe9ee9fc"}
May 25 13:28:03.705682 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:28:03.705678    2557 scope.go:117] "RemoveContainer" containerID="028abd1df64aaed5d32a63d5e34ad67ec195b9fc365e2224ea750477a06fb79f"
May 25 13:28:03.714477 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:28:03.714459    2557 scope.go:117] "RemoveContainer" containerID="93d67d53ec073097a9722982e7db613088317a1d3e2b1f29fa9e41629477c114"
May 25 13:28:03.722175 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:28:03.722154    2557 scope.go:117] "RemoveContainer" containerID="6b7c8c0de4f2b8d099a997c3f29ac3f757cb2eadffc9019365d8117aeb8ff241"
May 25 13:28:03.729527 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:28:03.729499    2557 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-7db8dcsfcx"]
May 25 13:28:03.730624 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:28:03.730609    2557 scope.go:117] "RemoveContainer" containerID="028abd1df64aaed5d32a63d5e34ad67ec195b9fc365e2224ea750477a06fb79f"
May 25 13:28:03.730963 ip-10-0-143-235 kubenswrapper[2557]: E0525 13:28:03.730931    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"028abd1df64aaed5d32a63d5e34ad67ec195b9fc365e2224ea750477a06fb79f\": container with ID starting with 028abd1df64aaed5d32a63d5e34ad67ec195b9fc365e2224ea750477a06fb79f not found: ID does not exist" containerID="028abd1df64aaed5d32a63d5e34ad67ec195b9fc365e2224ea750477a06fb79f"
May 25 13:28:03.731047 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:28:03.730976    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"028abd1df64aaed5d32a63d5e34ad67ec195b9fc365e2224ea750477a06fb79f"} err="failed to get container status \"028abd1df64aaed5d32a63d5e34ad67ec195b9fc365e2224ea750477a06fb79f\": rpc error: code = NotFound desc = could not find container \"028abd1df64aaed5d32a63d5e34ad67ec195b9fc365e2224ea750477a06fb79f\": container with ID starting with 028abd1df64aaed5d32a63d5e34ad67ec195b9fc365e2224ea750477a06fb79f not found: ID does not exist"
May 25 13:28:03.731047 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:28:03.731002    2557 scope.go:117] "RemoveContainer" containerID="93d67d53ec073097a9722982e7db613088317a1d3e2b1f29fa9e41629477c114"
May 25 13:28:03.731728 ip-10-0-143-235 kubenswrapper[2557]: E0525 13:28:03.731674    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93d67d53ec073097a9722982e7db613088317a1d3e2b1f29fa9e41629477c114\": container with ID starting with 93d67d53ec073097a9722982e7db613088317a1d3e2b1f29fa9e41629477c114 not found: ID does not exist" containerID="93d67d53ec073097a9722982e7db613088317a1d3e2b1f29fa9e41629477c114"
May 25 13:28:03.731728 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:28:03.731712    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93d67d53ec073097a9722982e7db613088317a1d3e2b1f29fa9e41629477c114"} err="failed to get container status \"93d67d53ec073097a9722982e7db613088317a1d3e2b1f29fa9e41629477c114\": rpc error: code = NotFound desc = could not find container \"93d67d53ec073097a9722982e7db613088317a1d3e2b1f29fa9e41629477c114\": container with ID starting with 93d67d53ec073097a9722982e7db613088317a1d3e2b1f29fa9e41629477c114 not found: ID does not exist"
May 25 13:28:03.731889 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:28:03.731755    2557 scope.go:117] "RemoveContainer" containerID="6b7c8c0de4f2b8d099a997c3f29ac3f757cb2eadffc9019365d8117aeb8ff241"
May 25 13:28:03.732336 ip-10-0-143-235 kubenswrapper[2557]: E0525 13:28:03.732287    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b7c8c0de4f2b8d099a997c3f29ac3f757cb2eadffc9019365d8117aeb8ff241\": container with ID starting with 6b7c8c0de4f2b8d099a997c3f29ac3f757cb2eadffc9019365d8117aeb8ff241 not found: ID does not exist" containerID="6b7c8c0de4f2b8d099a997c3f29ac3f757cb2eadffc9019365d8117aeb8ff241"
May 25 13:28:03.732336 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:28:03.732319    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b7c8c0de4f2b8d099a997c3f29ac3f757cb2eadffc9019365d8117aeb8ff241"} err="failed to get container status \"6b7c8c0de4f2b8d099a997c3f29ac3f757cb2eadffc9019365d8117aeb8ff241\": rpc error: code = NotFound desc = could not find container \"6b7c8c0de4f2b8d099a997c3f29ac3f757cb2eadffc9019365d8117aeb8ff241\": container with ID starting with 6b7c8c0de4f2b8d099a997c3f29ac3f757cb2eadffc9019365d8117aeb8ff241 not found: ID does not exist"
May 25 13:28:03.734371 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:28:03.734337    2557 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-7db8dcsfcx"]
May 25 13:28:04.154777 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:28:04.152954    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cacceccb-1084-4014-827d-e6ce1e92f795" path="/var/lib/kubelet/pods/cacceccb-1084-4014-827d-e6ce1e92f795/volumes"
May 25 13:28:08.965559 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:28:08.965523    2557 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-7474554f64w8"]
May 25 13:28:08.965949 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:28:08.965935    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cacceccb-1084-4014-827d-e6ce1e92f795" containerName="tokenizer"
May 25 13:28:08.965991 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:28:08.965950    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="cacceccb-1084-4014-827d-e6ce1e92f795" containerName="tokenizer"
May 25 13:28:08.965991 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:28:08.965960    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cacceccb-1084-4014-827d-e6ce1e92f795" containerName="main"
May 25 13:28:08.965991 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:28:08.965966    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="cacceccb-1084-4014-827d-e6ce1e92f795" containerName="main"
May 25 13:28:08.965991 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:28:08.965988    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cacceccb-1084-4014-827d-e6ce1e92f795" containerName="storage-initializer"
May 25 13:28:08.966102 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:28:08.965994    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="cacceccb-1084-4014-827d-e6ce1e92f795" containerName="storage-initializer"
May 25 13:28:08.966102 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:28:08.966048    2557 memory_manager.go:356] "RemoveStaleState removing state" podUID="cacceccb-1084-4014-827d-e6ce1e92f795" containerName="main"
May 25 13:28:08.966102 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:28:08.966057    2557 memory_manager.go:356] "RemoveStaleState removing state" podUID="cacceccb-1084-4014-827d-e6ce1e92f795" containerName="tokenizer"
May 25 13:28:08.968274 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:28:08.968256    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-7474554f64w8"
May 25 13:28:08.972414 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:28:08.972391    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-ha-replicas-test-epp-sa-dockercfg-kv9g7\""
May 25 13:28:08.972541 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:28:08.972413    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-ha-replicas-test-kserve-self-signed-certs\""
May 25 13:28:08.984496 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:28:08.984461    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-7474554f64w8"]
May 25 13:28:09.077272 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:28:09.077233    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/61494f9f-ab7e-4ad4-80a9-d071132ed511-tokenizer-tmp\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-7474554f64w8\" (UID: \"61494f9f-ab7e-4ad4-80a9-d071132ed511\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-7474554f64w8"
May 25 13:28:09.077522 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:28:09.077288    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/61494f9f-ab7e-4ad4-80a9-d071132ed511-tokenizer-cache\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-7474554f64w8\" (UID: \"61494f9f-ab7e-4ad4-80a9-d071132ed511\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-7474554f64w8"
May 25 13:28:09.077522 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:28:09.077314    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61494f9f-ab7e-4ad4-80a9-d071132ed511-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-7474554f64w8\" (UID: \"61494f9f-ab7e-4ad4-80a9-d071132ed511\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-7474554f64w8"
May 25 13:28:09.077522 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:28:09.077428    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/61494f9f-ab7e-4ad4-80a9-d071132ed511-tokenizer-uds\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-7474554f64w8\" (UID: \"61494f9f-ab7e-4ad4-80a9-d071132ed511\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-7474554f64w8"
May 25 13:28:09.077522 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:28:09.077470    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/61494f9f-ab7e-4ad4-80a9-d071132ed511-kserve-provision-location\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-7474554f64w8\" (UID: \"61494f9f-ab7e-4ad4-80a9-d071132ed511\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-7474554f64w8"
May 25 13:28:09.077719 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:28:09.077586    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hn9q\" (UniqueName: \"kubernetes.io/projected/61494f9f-ab7e-4ad4-80a9-d071132ed511-kube-api-access-5hn9q\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-7474554f64w8\" (UID: \"61494f9f-ab7e-4ad4-80a9-d071132ed511\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-7474554f64w8"
May 25 13:28:09.178967 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:28:09.178853    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/61494f9f-ab7e-4ad4-80a9-d071132ed511-tokenizer-cache\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-7474554f64w8\" (UID: \"61494f9f-ab7e-4ad4-80a9-d071132ed511\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-7474554f64w8"
May 25 13:28:09.178967 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:28:09.178909    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61494f9f-ab7e-4ad4-80a9-d071132ed511-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-7474554f64w8\" (UID: \"61494f9f-ab7e-4ad4-80a9-d071132ed511\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-7474554f64w8"
May 25 13:28:09.178967 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:28:09.178944    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/61494f9f-ab7e-4ad4-80a9-d071132ed511-tokenizer-uds\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-7474554f64w8\" (UID: \"61494f9f-ab7e-4ad4-80a9-d071132ed511\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-7474554f64w8"
May 25 13:28:09.179258 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:28:09.178973    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/61494f9f-ab7e-4ad4-80a9-d071132ed511-kserve-provision-location\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-7474554f64w8\" (UID: \"61494f9f-ab7e-4ad4-80a9-d071132ed511\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-7474554f64w8"
May 25 13:28:09.179258 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:28:09.179149    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-5hn9q\" (UniqueName: \"kubernetes.io/projected/61494f9f-ab7e-4ad4-80a9-d071132ed511-kube-api-access-5hn9q\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-7474554f64w8\" (UID: \"61494f9f-ab7e-4ad4-80a9-d071132ed511\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-7474554f64w8"
May 25 13:28:09.179398 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:28:09.179252    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/61494f9f-ab7e-4ad4-80a9-d071132ed511-tokenizer-tmp\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-7474554f64w8\" (UID: \"61494f9f-ab7e-4ad4-80a9-d071132ed511\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-7474554f64w8"
May 25 13:28:09.179398 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:28:09.179311    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/61494f9f-ab7e-4ad4-80a9-d071132ed511-tokenizer-cache\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-7474554f64w8\" (UID: \"61494f9f-ab7e-4ad4-80a9-d071132ed511\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-7474554f64w8"
May 25 13:28:09.179398 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:28:09.179378    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/61494f9f-ab7e-4ad4-80a9-d071132ed511-tokenizer-uds\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-7474554f64w8\" (UID: \"61494f9f-ab7e-4ad4-80a9-d071132ed511\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-7474554f64w8"
May 25 13:28:09.179530 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:28:09.179429    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/61494f9f-ab7e-4ad4-80a9-d071132ed511-kserve-provision-location\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-7474554f64w8\" (UID: \"61494f9f-ab7e-4ad4-80a9-d071132ed511\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-7474554f64w8"
May 25 13:28:09.179594 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:28:09.179575    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/61494f9f-ab7e-4ad4-80a9-d071132ed511-tokenizer-tmp\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-7474554f64w8\" (UID: \"61494f9f-ab7e-4ad4-80a9-d071132ed511\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-7474554f64w8"
May 25 13:28:09.181410 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:28:09.181392    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61494f9f-ab7e-4ad4-80a9-d071132ed511-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-7474554f64w8\" (UID: \"61494f9f-ab7e-4ad4-80a9-d071132ed511\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-7474554f64w8"
May 25 13:28:09.200574 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:28:09.200540    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hn9q\" (UniqueName: \"kubernetes.io/projected/61494f9f-ab7e-4ad4-80a9-d071132ed511-kube-api-access-5hn9q\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-7474554f64w8\" (UID: \"61494f9f-ab7e-4ad4-80a9-d071132ed511\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-7474554f64w8"
May 25 13:28:09.278430 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:28:09.278390    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-7474554f64w8"
May 25 13:28:09.422501 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:28:09.422473    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-7474554f64w8"]
May 25 13:28:09.424685 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:28:09.424660    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61494f9f_ab7e_4ad4_80a9_d071132ed511.slice/crio-43926d1f51df0ad6768c485a567823c1e65274e0341ff70d40de34f39e915747 WatchSource:0}: Error finding container 43926d1f51df0ad6768c485a567823c1e65274e0341ff70d40de34f39e915747: Status 404 returned error can't find the container with id 43926d1f51df0ad6768c485a567823c1e65274e0341ff70d40de34f39e915747
May 25 13:28:09.729947 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:28:09.729860    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-7474554f64w8" event={"ID":"61494f9f-ab7e-4ad4-80a9-d071132ed511","Type":"ContainerStarted","Data":"de49319ff15a2f52c4c178439f0799aa379498a8d5dc60fdbc84fdbb7e1fcf8d"}
May 25 13:28:09.729947 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:28:09.729897    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-7474554f64w8" event={"ID":"61494f9f-ab7e-4ad4-80a9-d071132ed511","Type":"ContainerStarted","Data":"43926d1f51df0ad6768c485a567823c1e65274e0341ff70d40de34f39e915747"}
May 25 13:28:10.734454 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:28:10.734415    2557 generic.go:358] "Generic (PLEG): container finished" podID="61494f9f-ab7e-4ad4-80a9-d071132ed511" containerID="de49319ff15a2f52c4c178439f0799aa379498a8d5dc60fdbc84fdbb7e1fcf8d" exitCode=0
May 25 13:28:10.734912 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:28:10.734468    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-7474554f64w8" event={"ID":"61494f9f-ab7e-4ad4-80a9-d071132ed511","Type":"ContainerDied","Data":"de49319ff15a2f52c4c178439f0799aa379498a8d5dc60fdbc84fdbb7e1fcf8d"}
May 25 13:28:11.739891 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:28:11.739856    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-7474554f64w8" event={"ID":"61494f9f-ab7e-4ad4-80a9-d071132ed511","Type":"ContainerStarted","Data":"d401c15d4fac54f46f32568aea8d48e4df5ec3f083aafccdaa7e9251fe9fad20"}
May 25 13:28:11.739891 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:28:11.739892    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-7474554f64w8" event={"ID":"61494f9f-ab7e-4ad4-80a9-d071132ed511","Type":"ContainerStarted","Data":"53790f958ee1d8f0aba7eb1983851e061986c8e9d6eb7fe04aa2291adafb6773"}
May 25 13:28:11.740324 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:28:11.739970    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-7474554f64w8"
May 25 13:28:11.764108 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:28:11.764044    2557 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-7474554f64w8" podStartSLOduration=3.7640202499999997 podStartE2EDuration="3.76402025s" podCreationTimestamp="2026-05-25 13:28:08 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 13:28:11.762038495 +0000 UTC m=+1344.225054649" watchObservedRunningTime="2026-05-25 13:28:11.76402025 +0000 UTC m=+1344.227036408"
May 25 13:28:19.279005 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:28:19.278956    2557 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-7474554f64w8"
May 25 13:28:19.279005 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:28:19.279015    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-7474554f64w8"
May 25 13:28:19.281622 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:28:19.281591    2557 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-7474554f64w8"
May 25 13:28:19.776266 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:28:19.776235    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-7474554f64w8"
May 25 13:28:21.696784 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:28:21.696752    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schexf6gg"
May 25 13:28:40.780969 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:28:40.780933    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-7474554f64w8"
May 25 13:30:41.003301 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:30:41.003263    2557 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-7474554f64w8"]
May 25 13:30:41.003830 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:30:41.003602    2557 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-7474554f64w8" podUID="61494f9f-ab7e-4ad4-80a9-d071132ed511" containerName="main" containerID="cri-o://53790f958ee1d8f0aba7eb1983851e061986c8e9d6eb7fe04aa2291adafb6773" gracePeriod=30
May 25 13:30:41.003830 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:30:41.003639    2557 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-7474554f64w8" podUID="61494f9f-ab7e-4ad4-80a9-d071132ed511" containerName="tokenizer" containerID="cri-o://d401c15d4fac54f46f32568aea8d48e4df5ec3f083aafccdaa7e9251fe9fad20" gracePeriod=30
May 25 13:30:41.302800 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:30:41.302767    2557 generic.go:358] "Generic (PLEG): container finished" podID="61494f9f-ab7e-4ad4-80a9-d071132ed511" containerID="53790f958ee1d8f0aba7eb1983851e061986c8e9d6eb7fe04aa2291adafb6773" exitCode=0
May 25 13:30:41.302987 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:30:41.302848    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-7474554f64w8" event={"ID":"61494f9f-ab7e-4ad4-80a9-d071132ed511","Type":"ContainerDied","Data":"53790f958ee1d8f0aba7eb1983851e061986c8e9d6eb7fe04aa2291adafb6773"}
May 25 13:30:42.309081 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:30:42.309050    2557 generic.go:358] "Generic (PLEG): container finished" podID="61494f9f-ab7e-4ad4-80a9-d071132ed511" containerID="d401c15d4fac54f46f32568aea8d48e4df5ec3f083aafccdaa7e9251fe9fad20" exitCode=0
May 25 13:30:42.309444 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:30:42.309124    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-7474554f64w8" event={"ID":"61494f9f-ab7e-4ad4-80a9-d071132ed511","Type":"ContainerDied","Data":"d401c15d4fac54f46f32568aea8d48e4df5ec3f083aafccdaa7e9251fe9fad20"}
May 25 13:30:42.360904 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:30:42.360877    2557 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-7474554f64w8"
May 25 13:30:42.407061 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:30:42.407015    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61494f9f-ab7e-4ad4-80a9-d071132ed511-tls-certs\") pod \"61494f9f-ab7e-4ad4-80a9-d071132ed511\" (UID: \"61494f9f-ab7e-4ad4-80a9-d071132ed511\") "
May 25 13:30:42.407260 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:30:42.407099    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/61494f9f-ab7e-4ad4-80a9-d071132ed511-tokenizer-uds\") pod \"61494f9f-ab7e-4ad4-80a9-d071132ed511\" (UID: \"61494f9f-ab7e-4ad4-80a9-d071132ed511\") "
May 25 13:30:42.407260 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:30:42.407141    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5hn9q\" (UniqueName: \"kubernetes.io/projected/61494f9f-ab7e-4ad4-80a9-d071132ed511-kube-api-access-5hn9q\") pod \"61494f9f-ab7e-4ad4-80a9-d071132ed511\" (UID: \"61494f9f-ab7e-4ad4-80a9-d071132ed511\") "
May 25 13:30:42.407260 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:30:42.407170    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/61494f9f-ab7e-4ad4-80a9-d071132ed511-tokenizer-cache\") pod \"61494f9f-ab7e-4ad4-80a9-d071132ed511\" (UID: \"61494f9f-ab7e-4ad4-80a9-d071132ed511\") "
May 25 13:30:42.407260 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:30:42.407195    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/61494f9f-ab7e-4ad4-80a9-d071132ed511-tokenizer-tmp\") pod \"61494f9f-ab7e-4ad4-80a9-d071132ed511\" (UID: \"61494f9f-ab7e-4ad4-80a9-d071132ed511\") "
May 25 13:30:42.407260 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:30:42.407238    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/61494f9f-ab7e-4ad4-80a9-d071132ed511-kserve-provision-location\") pod \"61494f9f-ab7e-4ad4-80a9-d071132ed511\" (UID: \"61494f9f-ab7e-4ad4-80a9-d071132ed511\") "
May 25 13:30:42.407556 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:30:42.407367    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61494f9f-ab7e-4ad4-80a9-d071132ed511-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "61494f9f-ab7e-4ad4-80a9-d071132ed511" (UID: "61494f9f-ab7e-4ad4-80a9-d071132ed511"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 13:30:42.407556 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:30:42.407485    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61494f9f-ab7e-4ad4-80a9-d071132ed511-tokenizer-cache" (OuterVolumeSpecName: "tokenizer-cache") pod "61494f9f-ab7e-4ad4-80a9-d071132ed511" (UID: "61494f9f-ab7e-4ad4-80a9-d071132ed511"). InnerVolumeSpecName "tokenizer-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 13:30:42.407556 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:30:42.407546    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61494f9f-ab7e-4ad4-80a9-d071132ed511-tokenizer-tmp" (OuterVolumeSpecName: "tokenizer-tmp") pod "61494f9f-ab7e-4ad4-80a9-d071132ed511" (UID: "61494f9f-ab7e-4ad4-80a9-d071132ed511"). InnerVolumeSpecName "tokenizer-tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 13:30:42.407711 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:30:42.407633    2557 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/61494f9f-ab7e-4ad4-80a9-d071132ed511-tokenizer-uds\") on node \"ip-10-0-143-235.ec2.internal\" DevicePath \"\""
May 25 13:30:42.407711 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:30:42.407653    2557 reconciler_common.go:299] "Volume detached for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/61494f9f-ab7e-4ad4-80a9-d071132ed511-tokenizer-cache\") on node \"ip-10-0-143-235.ec2.internal\" DevicePath \"\""
May 25 13:30:42.407711 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:30:42.407666    2557 reconciler_common.go:299] "Volume detached for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/61494f9f-ab7e-4ad4-80a9-d071132ed511-tokenizer-tmp\") on node \"ip-10-0-143-235.ec2.internal\" DevicePath \"\""
May 25 13:30:42.408105 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:30:42.408077    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61494f9f-ab7e-4ad4-80a9-d071132ed511-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "61494f9f-ab7e-4ad4-80a9-d071132ed511" (UID: "61494f9f-ab7e-4ad4-80a9-d071132ed511"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 13:30:42.409256 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:30:42.409232    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/61494f9f-ab7e-4ad4-80a9-d071132ed511-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "61494f9f-ab7e-4ad4-80a9-d071132ed511" (UID: "61494f9f-ab7e-4ad4-80a9-d071132ed511"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 13:30:42.409376 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:30:42.409296    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61494f9f-ab7e-4ad4-80a9-d071132ed511-kube-api-access-5hn9q" (OuterVolumeSpecName: "kube-api-access-5hn9q") pod "61494f9f-ab7e-4ad4-80a9-d071132ed511" (UID: "61494f9f-ab7e-4ad4-80a9-d071132ed511"). InnerVolumeSpecName "kube-api-access-5hn9q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 13:30:42.509169 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:30:42.509081    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61494f9f-ab7e-4ad4-80a9-d071132ed511-tls-certs\") on node \"ip-10-0-143-235.ec2.internal\" DevicePath \"\""
May 25 13:30:42.509169 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:30:42.509113    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5hn9q\" (UniqueName: \"kubernetes.io/projected/61494f9f-ab7e-4ad4-80a9-d071132ed511-kube-api-access-5hn9q\") on node \"ip-10-0-143-235.ec2.internal\" DevicePath \"\""
May 25 13:30:42.509169 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:30:42.509128    2557 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/61494f9f-ab7e-4ad4-80a9-d071132ed511-kserve-provision-location\") on node \"ip-10-0-143-235.ec2.internal\" DevicePath \"\""
May 25 13:30:43.313999 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:30:43.313966    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-7474554f64w8" event={"ID":"61494f9f-ab7e-4ad4-80a9-d071132ed511","Type":"ContainerDied","Data":"43926d1f51df0ad6768c485a567823c1e65274e0341ff70d40de34f39e915747"}
May 25 13:30:43.313999 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:30:43.313984    2557 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-7474554f64w8"
May 25 13:30:43.314486 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:30:43.314011    2557 scope.go:117] "RemoveContainer" containerID="d401c15d4fac54f46f32568aea8d48e4df5ec3f083aafccdaa7e9251fe9fad20"
May 25 13:30:43.323405 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:30:43.323385    2557 scope.go:117] "RemoveContainer" containerID="53790f958ee1d8f0aba7eb1983851e061986c8e9d6eb7fe04aa2291adafb6773"
May 25 13:30:43.331179 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:30:43.331161    2557 scope.go:117] "RemoveContainer" containerID="de49319ff15a2f52c4c178439f0799aa379498a8d5dc60fdbc84fdbb7e1fcf8d"
May 25 13:30:43.337391 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:30:43.337364    2557 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-7474554f64w8"]
May 25 13:30:43.343245 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:30:43.343220    2557 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-7474554f64w8"]
May 25 13:30:44.150583 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:30:44.150547    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="61494f9f-ab7e-4ad4-80a9-d071132ed511" path="/var/lib/kubelet/pods/61494f9f-ab7e-4ad4-80a9-d071132ed511/volumes"
May 25 13:30:48.146476 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:30:48.146441    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vqnvc_d21bdf8b-d0ab-48f2-8736-54e92947dd53/ovn-acl-logging/0.log"
May 25 13:30:48.151571 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:30:48.151547    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vqnvc_d21bdf8b-d0ab-48f2-8736-54e92947dd53/ovn-acl-logging/0.log"
May 25 13:31:15.079442 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:31:15.079327    2557 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schexf6gg"]
May 25 13:31:15.080029 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:31:15.079706    2557 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schexf6gg" podUID="e6b7779d-d70f-4a73-960e-9b1e13044b28" containerName="main" containerID="cri-o://5241c281d3792568747e2f92664d979fbe725f0c426513177a459f03d05ef10b" gracePeriod=30
May 25 13:31:15.080029 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:31:15.079739    2557 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schexf6gg" podUID="e6b7779d-d70f-4a73-960e-9b1e13044b28" containerName="tokenizer" containerID="cri-o://3c302ef84d4cd42709f164d26a7090aab9ced5539eb30d868ae91555e0463355" gracePeriod=30
May 25 13:31:15.431137 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:31:15.431039    2557 generic.go:358] "Generic (PLEG): container finished" podID="e6b7779d-d70f-4a73-960e-9b1e13044b28" containerID="5241c281d3792568747e2f92664d979fbe725f0c426513177a459f03d05ef10b" exitCode=0
May 25 13:31:15.431137 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:31:15.431113    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schexf6gg" event={"ID":"e6b7779d-d70f-4a73-960e-9b1e13044b28","Type":"ContainerDied","Data":"5241c281d3792568747e2f92664d979fbe725f0c426513177a459f03d05ef10b"}
May 25 13:31:16.435957 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:31:16.435927    2557 generic.go:358] "Generic (PLEG): container finished" podID="e6b7779d-d70f-4a73-960e-9b1e13044b28" containerID="3c302ef84d4cd42709f164d26a7090aab9ced5539eb30d868ae91555e0463355" exitCode=0
May 25 13:31:16.436286 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:31:16.435965    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schexf6gg" event={"ID":"e6b7779d-d70f-4a73-960e-9b1e13044b28","Type":"ContainerDied","Data":"3c302ef84d4cd42709f164d26a7090aab9ced5539eb30d868ae91555e0463355"}
May 25 13:31:16.452090 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:31:16.452066    2557 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schexf6gg"
May 25 13:31:16.632435 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:31:16.632398    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6b7779d-d70f-4a73-960e-9b1e13044b28-tokenizer-cache\") pod \"e6b7779d-d70f-4a73-960e-9b1e13044b28\" (UID: \"e6b7779d-d70f-4a73-960e-9b1e13044b28\") "
May 25 13:31:16.632649 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:31:16.632493    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e6b7779d-d70f-4a73-960e-9b1e13044b28-kserve-provision-location\") pod \"e6b7779d-d70f-4a73-960e-9b1e13044b28\" (UID: \"e6b7779d-d70f-4a73-960e-9b1e13044b28\") "
May 25 13:31:16.632649 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:31:16.632522    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tgl8n\" (UniqueName: \"kubernetes.io/projected/e6b7779d-d70f-4a73-960e-9b1e13044b28-kube-api-access-tgl8n\") pod \"e6b7779d-d70f-4a73-960e-9b1e13044b28\" (UID: \"e6b7779d-d70f-4a73-960e-9b1e13044b28\") "
May 25 13:31:16.632649 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:31:16.632545    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/e6b7779d-d70f-4a73-960e-9b1e13044b28-tokenizer-tmp\") pod \"e6b7779d-d70f-4a73-960e-9b1e13044b28\" (UID: \"e6b7779d-d70f-4a73-960e-9b1e13044b28\") "
May 25 13:31:16.632649 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:31:16.632590    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6b7779d-d70f-4a73-960e-9b1e13044b28-tls-certs\") pod \"e6b7779d-d70f-4a73-960e-9b1e13044b28\" (UID: \"e6b7779d-d70f-4a73-960e-9b1e13044b28\") "
May 25 13:31:16.632841 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:31:16.632688    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/e6b7779d-d70f-4a73-960e-9b1e13044b28-tokenizer-uds\") pod \"e6b7779d-d70f-4a73-960e-9b1e13044b28\" (UID: \"e6b7779d-d70f-4a73-960e-9b1e13044b28\") "
May 25 13:31:16.632841 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:31:16.632743    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6b7779d-d70f-4a73-960e-9b1e13044b28-tokenizer-cache" (OuterVolumeSpecName: "tokenizer-cache") pod "e6b7779d-d70f-4a73-960e-9b1e13044b28" (UID: "e6b7779d-d70f-4a73-960e-9b1e13044b28"). InnerVolumeSpecName "tokenizer-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 13:31:16.632964 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:31:16.632891    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6b7779d-d70f-4a73-960e-9b1e13044b28-tokenizer-tmp" (OuterVolumeSpecName: "tokenizer-tmp") pod "e6b7779d-d70f-4a73-960e-9b1e13044b28" (UID: "e6b7779d-d70f-4a73-960e-9b1e13044b28"). InnerVolumeSpecName "tokenizer-tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 13:31:16.633027 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:31:16.632975    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6b7779d-d70f-4a73-960e-9b1e13044b28-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "e6b7779d-d70f-4a73-960e-9b1e13044b28" (UID: "e6b7779d-d70f-4a73-960e-9b1e13044b28"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 13:31:16.633027 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:31:16.633009    2557 reconciler_common.go:299] "Volume detached for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6b7779d-d70f-4a73-960e-9b1e13044b28-tokenizer-cache\") on node \"ip-10-0-143-235.ec2.internal\" DevicePath \"\""
May 25 13:31:16.633289 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:31:16.633268    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6b7779d-d70f-4a73-960e-9b1e13044b28-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e6b7779d-d70f-4a73-960e-9b1e13044b28" (UID: "e6b7779d-d70f-4a73-960e-9b1e13044b28"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 13:31:16.634707 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:31:16.634687    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6b7779d-d70f-4a73-960e-9b1e13044b28-kube-api-access-tgl8n" (OuterVolumeSpecName: "kube-api-access-tgl8n") pod "e6b7779d-d70f-4a73-960e-9b1e13044b28" (UID: "e6b7779d-d70f-4a73-960e-9b1e13044b28"). InnerVolumeSpecName "kube-api-access-tgl8n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 13:31:16.634765 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:31:16.634716    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e6b7779d-d70f-4a73-960e-9b1e13044b28-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e6b7779d-d70f-4a73-960e-9b1e13044b28" (UID: "e6b7779d-d70f-4a73-960e-9b1e13044b28"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 13:31:16.733881 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:31:16.733836    2557 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/e6b7779d-d70f-4a73-960e-9b1e13044b28-tokenizer-uds\") on node \"ip-10-0-143-235.ec2.internal\" DevicePath \"\""
May 25 13:31:16.733881 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:31:16.733872    2557 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e6b7779d-d70f-4a73-960e-9b1e13044b28-kserve-provision-location\") on node \"ip-10-0-143-235.ec2.internal\" DevicePath \"\""
May 25 13:31:16.733881 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:31:16.733882    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tgl8n\" (UniqueName: \"kubernetes.io/projected/e6b7779d-d70f-4a73-960e-9b1e13044b28-kube-api-access-tgl8n\") on node \"ip-10-0-143-235.ec2.internal\" DevicePath \"\""
May 25 13:31:16.733881 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:31:16.733891    2557 reconciler_common.go:299] "Volume detached for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/e6b7779d-d70f-4a73-960e-9b1e13044b28-tokenizer-tmp\") on node \"ip-10-0-143-235.ec2.internal\" DevicePath \"\""
May 25 13:31:16.734159 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:31:16.733901    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6b7779d-d70f-4a73-960e-9b1e13044b28-tls-certs\") on node \"ip-10-0-143-235.ec2.internal\" DevicePath \"\""
May 25 13:31:17.441539 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:31:17.441503    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schexf6gg" event={"ID":"e6b7779d-d70f-4a73-960e-9b1e13044b28","Type":"ContainerDied","Data":"68b40df2cefa53accf5e1d5b24edeeb98c2b02eaf7a8a0104cbe850d89b521e7"}
May 25 13:31:17.442027 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:31:17.441547    2557 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schexf6gg"
May 25 13:31:17.442027 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:31:17.441552    2557 scope.go:117] "RemoveContainer" containerID="3c302ef84d4cd42709f164d26a7090aab9ced5539eb30d868ae91555e0463355"
May 25 13:31:17.451184 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:31:17.451161    2557 scope.go:117] "RemoveContainer" containerID="5241c281d3792568747e2f92664d979fbe725f0c426513177a459f03d05ef10b"
May 25 13:31:17.461710 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:31:17.461686    2557 scope.go:117] "RemoveContainer" containerID="81057c4d50a1166664e47323146c5e63f19f800b13796a58e8dda4db08c1f271"
May 25 13:31:17.466036 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:31:17.466008    2557 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schexf6gg"]
May 25 13:31:17.473458 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:31:17.473430    2557 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schexf6gg"]
May 25 13:31:18.156023 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:31:18.155983    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e6b7779d-d70f-4a73-960e-9b1e13044b28" path="/var/lib/kubelet/pods/e6b7779d-d70f-4a73-960e-9b1e13044b28/volumes"
May 25 13:31:23.517625 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:31:23.517578    2557 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-5798769bxrd2g"]
May 25 13:31:23.518064 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:31:23.518047    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e6b7779d-d70f-4a73-960e-9b1e13044b28" containerName="storage-initializer"
May 25 13:31:23.518112 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:31:23.518066    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="e6b7779d-d70f-4a73-960e-9b1e13044b28" containerName="storage-initializer"
May 25 13:31:23.518112 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:31:23.518080    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="61494f9f-ab7e-4ad4-80a9-d071132ed511" containerName="tokenizer"
May 25 13:31:23.518112 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:31:23.518085    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="61494f9f-ab7e-4ad4-80a9-d071132ed511" containerName="tokenizer"
May 25 13:31:23.518112 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:31:23.518102    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="61494f9f-ab7e-4ad4-80a9-d071132ed511" containerName="main"
May 25 13:31:23.518112 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:31:23.518107    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="61494f9f-ab7e-4ad4-80a9-d071132ed511" containerName="main"
May 25 13:31:23.518255 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:31:23.518115    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e6b7779d-d70f-4a73-960e-9b1e13044b28" containerName="main"
May 25 13:31:23.518255 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:31:23.518121    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="e6b7779d-d70f-4a73-960e-9b1e13044b28" containerName="main"
May 25 13:31:23.518255 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:31:23.518126    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="61494f9f-ab7e-4ad4-80a9-d071132ed511" containerName="storage-initializer"
May 25 13:31:23.518255 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:31:23.518131    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="61494f9f-ab7e-4ad4-80a9-d071132ed511" containerName="storage-initializer"
May 25 13:31:23.518255 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:31:23.518139    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e6b7779d-d70f-4a73-960e-9b1e13044b28" containerName="tokenizer"
May 25 13:31:23.518255 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:31:23.518143    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="e6b7779d-d70f-4a73-960e-9b1e13044b28" containerName="tokenizer"
May 25 13:31:23.518255 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:31:23.518203    2557 memory_manager.go:356] "RemoveStaleState removing state" podUID="61494f9f-ab7e-4ad4-80a9-d071132ed511" containerName="main"
May 25 13:31:23.518255 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:31:23.518212    2557 memory_manager.go:356] "RemoveStaleState removing state" podUID="e6b7779d-d70f-4a73-960e-9b1e13044b28" containerName="tokenizer"
May 25 13:31:23.518255 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:31:23.518217    2557 memory_manager.go:356] "RemoveStaleState removing state" podUID="61494f9f-ab7e-4ad4-80a9-d071132ed511" containerName="tokenizer"
May 25 13:31:23.518255 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:31:23.518226    2557 memory_manager.go:356] "RemoveStaleState removing state" podUID="e6b7779d-d70f-4a73-960e-9b1e13044b28" containerName="main"
May 25 13:31:23.523206 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:31:23.523174    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-5798769bxrd2g"
May 25 13:31:23.526058 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:31:23.526029    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
May 25 13:31:23.526558 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:31:23.526179    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"custom-route-timeout-test-kserve-self-signed-certs\""
May 25 13:31:23.526558 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:31:23.526194    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
May 25 13:31:23.527177 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:31:23.527161    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"custom-route-timeout-test-epp-sa-dockercfg-mh2xw\""
May 25 13:31:23.527271 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:31:23.527248    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-8blwv\""
May 25 13:31:23.539372 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:31:23.539327    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-5798769bxrd2g"]
May 25 13:31:23.590055 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:31:23.590017    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bsmx2\" (UniqueName: \"kubernetes.io/projected/f02cb33c-c306-4ceb-819f-bd151bd7e3c7-kube-api-access-bsmx2\") pod \"custom-route-timeout-test-kserve-router-scheduler-5798769bxrd2g\" (UID: \"f02cb33c-c306-4ceb-819f-bd151bd7e3c7\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-5798769bxrd2g"
May 25 13:31:23.590244 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:31:23.590068    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f02cb33c-c306-4ceb-819f-bd151bd7e3c7-tls-certs\") pod \"custom-route-timeout-test-kserve-router-scheduler-5798769bxrd2g\" (UID: \"f02cb33c-c306-4ceb-819f-bd151bd7e3c7\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-5798769bxrd2g"
May 25 13:31:23.590244 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:31:23.590111    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/f02cb33c-c306-4ceb-819f-bd151bd7e3c7-tokenizer-tmp\") pod \"custom-route-timeout-test-kserve-router-scheduler-5798769bxrd2g\" (UID: \"f02cb33c-c306-4ceb-819f-bd151bd7e3c7\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-5798769bxrd2g"
May 25 13:31:23.590244 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:31:23.590147    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/f02cb33c-c306-4ceb-819f-bd151bd7e3c7-tokenizer-cache\") pod \"custom-route-timeout-test-kserve-router-scheduler-5798769bxrd2g\" (UID: \"f02cb33c-c306-4ceb-819f-bd151bd7e3c7\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-5798769bxrd2g"
May 25 13:31:23.590244 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:31:23.590172    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f02cb33c-c306-4ceb-819f-bd151bd7e3c7-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-router-scheduler-5798769bxrd2g\" (UID: \"f02cb33c-c306-4ceb-819f-bd151bd7e3c7\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-5798769bxrd2g"
May 25 13:31:23.590244 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:31:23.590221    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/f02cb33c-c306-4ceb-819f-bd151bd7e3c7-tokenizer-uds\") pod \"custom-route-timeout-test-kserve-router-scheduler-5798769bxrd2g\" (UID: \"f02cb33c-c306-4ceb-819f-bd151bd7e3c7\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-5798769bxrd2g"
May 25 13:31:23.691017 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:31:23.690974    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/f02cb33c-c306-4ceb-819f-bd151bd7e3c7-tokenizer-uds\") pod \"custom-route-timeout-test-kserve-router-scheduler-5798769bxrd2g\" (UID: \"f02cb33c-c306-4ceb-819f-bd151bd7e3c7\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-5798769bxrd2g"
May 25 13:31:23.691198 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:31:23.691055    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-bsmx2\" (UniqueName: \"kubernetes.io/projected/f02cb33c-c306-4ceb-819f-bd151bd7e3c7-kube-api-access-bsmx2\") pod \"custom-route-timeout-test-kserve-router-scheduler-5798769bxrd2g\" (UID: \"f02cb33c-c306-4ceb-819f-bd151bd7e3c7\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-5798769bxrd2g"
May 25 13:31:23.691198 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:31:23.691095    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f02cb33c-c306-4ceb-819f-bd151bd7e3c7-tls-certs\") pod \"custom-route-timeout-test-kserve-router-scheduler-5798769bxrd2g\" (UID: \"f02cb33c-c306-4ceb-819f-bd151bd7e3c7\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-5798769bxrd2g"
May 25 13:31:23.691198 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:31:23.691118    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/f02cb33c-c306-4ceb-819f-bd151bd7e3c7-tokenizer-tmp\") pod \"custom-route-timeout-test-kserve-router-scheduler-5798769bxrd2g\" (UID: \"f02cb33c-c306-4ceb-819f-bd151bd7e3c7\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-5798769bxrd2g"
May 25 13:31:23.691198 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:31:23.691147    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/f02cb33c-c306-4ceb-819f-bd151bd7e3c7-tokenizer-cache\") pod \"custom-route-timeout-test-kserve-router-scheduler-5798769bxrd2g\" (UID: \"f02cb33c-c306-4ceb-819f-bd151bd7e3c7\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-5798769bxrd2g"
May 25 13:31:23.691198 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:31:23.691179    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f02cb33c-c306-4ceb-819f-bd151bd7e3c7-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-router-scheduler-5798769bxrd2g\" (UID: \"f02cb33c-c306-4ceb-819f-bd151bd7e3c7\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-5798769bxrd2g"
May 25 13:31:23.691494 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:31:23.691468    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/f02cb33c-c306-4ceb-819f-bd151bd7e3c7-tokenizer-uds\") pod \"custom-route-timeout-test-kserve-router-scheduler-5798769bxrd2g\" (UID: \"f02cb33c-c306-4ceb-819f-bd151bd7e3c7\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-5798769bxrd2g"
May 25 13:31:23.691562 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:31:23.691488    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/f02cb33c-c306-4ceb-819f-bd151bd7e3c7-tokenizer-cache\") pod \"custom-route-timeout-test-kserve-router-scheduler-5798769bxrd2g\" (UID: \"f02cb33c-c306-4ceb-819f-bd151bd7e3c7\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-5798769bxrd2g"
May 25 13:31:23.691562 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:31:23.691540    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/f02cb33c-c306-4ceb-819f-bd151bd7e3c7-tokenizer-tmp\") pod \"custom-route-timeout-test-kserve-router-scheduler-5798769bxrd2g\" (UID: \"f02cb33c-c306-4ceb-819f-bd151bd7e3c7\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-5798769bxrd2g"
May 25 13:31:23.691687 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:31:23.691659    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f02cb33c-c306-4ceb-819f-bd151bd7e3c7-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-router-scheduler-5798769bxrd2g\" (UID: \"f02cb33c-c306-4ceb-819f-bd151bd7e3c7\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-5798769bxrd2g"
May 25 13:31:23.693642 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:31:23.693623    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f02cb33c-c306-4ceb-819f-bd151bd7e3c7-tls-certs\") pod \"custom-route-timeout-test-kserve-router-scheduler-5798769bxrd2g\" (UID: \"f02cb33c-c306-4ceb-819f-bd151bd7e3c7\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-5798769bxrd2g"
May 25 13:31:23.699566 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:31:23.699538    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bsmx2\" (UniqueName: \"kubernetes.io/projected/f02cb33c-c306-4ceb-819f-bd151bd7e3c7-kube-api-access-bsmx2\") pod \"custom-route-timeout-test-kserve-router-scheduler-5798769bxrd2g\" (UID: \"f02cb33c-c306-4ceb-819f-bd151bd7e3c7\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-5798769bxrd2g"
May 25 13:31:23.835459 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:31:23.835414    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-5798769bxrd2g"
May 25 13:31:23.972417 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:31:23.972373    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-5798769bxrd2g"]
May 25 13:31:23.974819 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:31:23.974788    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf02cb33c_c306_4ceb_819f_bd151bd7e3c7.slice/crio-1a88d4ee0b171d39e3ef71b2bb5000d6a5e89ba871d87b16480525f450dc0486 WatchSource:0}: Error finding container 1a88d4ee0b171d39e3ef71b2bb5000d6a5e89ba871d87b16480525f450dc0486: Status 404 returned error can't find the container with id 1a88d4ee0b171d39e3ef71b2bb5000d6a5e89ba871d87b16480525f450dc0486
May 25 13:31:23.976697 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:31:23.976680    2557 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 25 13:31:24.469297 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:31:24.469260    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-5798769bxrd2g" event={"ID":"f02cb33c-c306-4ceb-819f-bd151bd7e3c7","Type":"ContainerStarted","Data":"761cef318e370319e1d8f71ddb62e934075db12a6c8c401fdd6c5e7c0daa2ae1"}
May 25 13:31:24.469297 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:31:24.469302    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-5798769bxrd2g" event={"ID":"f02cb33c-c306-4ceb-819f-bd151bd7e3c7","Type":"ContainerStarted","Data":"1a88d4ee0b171d39e3ef71b2bb5000d6a5e89ba871d87b16480525f450dc0486"}
May 25 13:31:25.473727 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:31:25.473683    2557 generic.go:358] "Generic (PLEG): container finished" podID="f02cb33c-c306-4ceb-819f-bd151bd7e3c7" containerID="761cef318e370319e1d8f71ddb62e934075db12a6c8c401fdd6c5e7c0daa2ae1" exitCode=0
May 25 13:31:25.474110 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:31:25.473773    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-5798769bxrd2g" event={"ID":"f02cb33c-c306-4ceb-819f-bd151bd7e3c7","Type":"ContainerDied","Data":"761cef318e370319e1d8f71ddb62e934075db12a6c8c401fdd6c5e7c0daa2ae1"}
May 25 13:31:26.479581 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:31:26.479536    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-5798769bxrd2g" event={"ID":"f02cb33c-c306-4ceb-819f-bd151bd7e3c7","Type":"ContainerStarted","Data":"2be2be0cc6672144eb66adefc155f28370419ec5467577a6349f0d671031baef"}
May 25 13:31:26.479581 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:31:26.479583    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-5798769bxrd2g" event={"ID":"f02cb33c-c306-4ceb-819f-bd151bd7e3c7","Type":"ContainerStarted","Data":"82a672b5dd56cb42d84865b8642f117b24160795a8fda712ce124b8de2f7a9d8"}
May 25 13:31:26.480021 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:31:26.479682    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-5798769bxrd2g"
May 25 13:31:26.505309 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:31:26.505258    2557 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-5798769bxrd2g" podStartSLOduration=3.5052434 podStartE2EDuration="3.5052434s" podCreationTimestamp="2026-05-25 13:31:23 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 13:31:26.50353804 +0000 UTC m=+1538.966554200" watchObservedRunningTime="2026-05-25 13:31:26.5052434 +0000 UTC m=+1538.968259556"
May 25 13:31:33.836531 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:31:33.836482    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-5798769bxrd2g"
May 25 13:31:33.836946 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:31:33.836548    2557 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-5798769bxrd2g"
May 25 13:31:33.839218 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:31:33.839190    2557 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-5798769bxrd2g"
May 25 13:31:34.509777 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:31:34.509741    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-5798769bxrd2g"
May 25 13:31:55.513765 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:31:55.513736    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-5798769bxrd2g"
May 25 13:35:48.178475 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:35:48.178447    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vqnvc_d21bdf8b-d0ab-48f2-8736-54e92947dd53/ovn-acl-logging/0.log"
May 25 13:35:48.184296 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:35:48.184276    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vqnvc_d21bdf8b-d0ab-48f2-8736-54e92947dd53/ovn-acl-logging/0.log"
May 25 13:36:09.485990 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:36:09.485955    2557 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-5798769bxrd2g"]
May 25 13:36:09.486580 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:36:09.486284    2557 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-5798769bxrd2g" podUID="f02cb33c-c306-4ceb-819f-bd151bd7e3c7" containerName="main" containerID="cri-o://82a672b5dd56cb42d84865b8642f117b24160795a8fda712ce124b8de2f7a9d8" gracePeriod=30
May 25 13:36:09.486580 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:36:09.486334    2557 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-5798769bxrd2g" podUID="f02cb33c-c306-4ceb-819f-bd151bd7e3c7" containerName="tokenizer" containerID="cri-o://2be2be0cc6672144eb66adefc155f28370419ec5467577a6349f0d671031baef" gracePeriod=30
May 25 13:36:10.516476 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:36:10.516438    2557 generic.go:358] "Generic (PLEG): container finished" podID="f02cb33c-c306-4ceb-819f-bd151bd7e3c7" containerID="82a672b5dd56cb42d84865b8642f117b24160795a8fda712ce124b8de2f7a9d8" exitCode=0
May 25 13:36:10.516864 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:36:10.516510    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-5798769bxrd2g" event={"ID":"f02cb33c-c306-4ceb-819f-bd151bd7e3c7","Type":"ContainerDied","Data":"82a672b5dd56cb42d84865b8642f117b24160795a8fda712ce124b8de2f7a9d8"}
May 25 13:36:10.846474 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:36:10.846444    2557 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-5798769bxrd2g"
May 25 13:36:10.920692 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:36:10.920657    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bsmx2\" (UniqueName: \"kubernetes.io/projected/f02cb33c-c306-4ceb-819f-bd151bd7e3c7-kube-api-access-bsmx2\") pod \"f02cb33c-c306-4ceb-819f-bd151bd7e3c7\" (UID: \"f02cb33c-c306-4ceb-819f-bd151bd7e3c7\") "
May 25 13:36:10.920882 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:36:10.920701    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/f02cb33c-c306-4ceb-819f-bd151bd7e3c7-tokenizer-uds\") pod \"f02cb33c-c306-4ceb-819f-bd151bd7e3c7\" (UID: \"f02cb33c-c306-4ceb-819f-bd151bd7e3c7\") "
May 25 13:36:10.920882 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:36:10.920724    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/f02cb33c-c306-4ceb-819f-bd151bd7e3c7-tokenizer-tmp\") pod \"f02cb33c-c306-4ceb-819f-bd151bd7e3c7\" (UID: \"f02cb33c-c306-4ceb-819f-bd151bd7e3c7\") "
May 25 13:36:10.920882 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:36:10.920747    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f02cb33c-c306-4ceb-819f-bd151bd7e3c7-kserve-provision-location\") pod \"f02cb33c-c306-4ceb-819f-bd151bd7e3c7\" (UID: \"f02cb33c-c306-4ceb-819f-bd151bd7e3c7\") "
May 25 13:36:10.920882 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:36:10.920778    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/f02cb33c-c306-4ceb-819f-bd151bd7e3c7-tokenizer-cache\") pod \"f02cb33c-c306-4ceb-819f-bd151bd7e3c7\" (UID: \"f02cb33c-c306-4ceb-819f-bd151bd7e3c7\") "
May 25 13:36:10.920882 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:36:10.920844    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f02cb33c-c306-4ceb-819f-bd151bd7e3c7-tls-certs\") pod \"f02cb33c-c306-4ceb-819f-bd151bd7e3c7\" (UID: \"f02cb33c-c306-4ceb-819f-bd151bd7e3c7\") "
May 25 13:36:10.921210 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:36:10.921045    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f02cb33c-c306-4ceb-819f-bd151bd7e3c7-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "f02cb33c-c306-4ceb-819f-bd151bd7e3c7" (UID: "f02cb33c-c306-4ceb-819f-bd151bd7e3c7"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 13:36:10.921210 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:36:10.921067    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f02cb33c-c306-4ceb-819f-bd151bd7e3c7-tokenizer-cache" (OuterVolumeSpecName: "tokenizer-cache") pod "f02cb33c-c306-4ceb-819f-bd151bd7e3c7" (UID: "f02cb33c-c306-4ceb-819f-bd151bd7e3c7"). InnerVolumeSpecName "tokenizer-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 13:36:10.921210 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:36:10.921119    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f02cb33c-c306-4ceb-819f-bd151bd7e3c7-tokenizer-tmp" (OuterVolumeSpecName: "tokenizer-tmp") pod "f02cb33c-c306-4ceb-819f-bd151bd7e3c7" (UID: "f02cb33c-c306-4ceb-819f-bd151bd7e3c7"). InnerVolumeSpecName "tokenizer-tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 13:36:10.921210 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:36:10.921166    2557 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/f02cb33c-c306-4ceb-819f-bd151bd7e3c7-tokenizer-uds\") on node \"ip-10-0-143-235.ec2.internal\" DevicePath \"\""
May 25 13:36:10.921210 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:36:10.921184    2557 reconciler_common.go:299] "Volume detached for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/f02cb33c-c306-4ceb-819f-bd151bd7e3c7-tokenizer-cache\") on node \"ip-10-0-143-235.ec2.internal\" DevicePath \"\""
May 25 13:36:10.921618 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:36:10.921595    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f02cb33c-c306-4ceb-819f-bd151bd7e3c7-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f02cb33c-c306-4ceb-819f-bd151bd7e3c7" (UID: "f02cb33c-c306-4ceb-819f-bd151bd7e3c7"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 13:36:10.922914 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:36:10.922887    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f02cb33c-c306-4ceb-819f-bd151bd7e3c7-kube-api-access-bsmx2" (OuterVolumeSpecName: "kube-api-access-bsmx2") pod "f02cb33c-c306-4ceb-819f-bd151bd7e3c7" (UID: "f02cb33c-c306-4ceb-819f-bd151bd7e3c7"). InnerVolumeSpecName "kube-api-access-bsmx2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 13:36:10.922914 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:36:10.922908    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f02cb33c-c306-4ceb-819f-bd151bd7e3c7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f02cb33c-c306-4ceb-819f-bd151bd7e3c7" (UID: "f02cb33c-c306-4ceb-819f-bd151bd7e3c7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 13:36:11.022292 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:36:11.022235    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f02cb33c-c306-4ceb-819f-bd151bd7e3c7-tls-certs\") on node \"ip-10-0-143-235.ec2.internal\" DevicePath \"\""
May 25 13:36:11.022292 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:36:11.022285    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bsmx2\" (UniqueName: \"kubernetes.io/projected/f02cb33c-c306-4ceb-819f-bd151bd7e3c7-kube-api-access-bsmx2\") on node \"ip-10-0-143-235.ec2.internal\" DevicePath \"\""
May 25 13:36:11.022292 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:36:11.022296    2557 reconciler_common.go:299] "Volume detached for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/f02cb33c-c306-4ceb-819f-bd151bd7e3c7-tokenizer-tmp\") on node \"ip-10-0-143-235.ec2.internal\" DevicePath \"\""
May 25 13:36:11.022292 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:36:11.022306    2557 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f02cb33c-c306-4ceb-819f-bd151bd7e3c7-kserve-provision-location\") on node \"ip-10-0-143-235.ec2.internal\" DevicePath \"\""
May 25 13:36:11.522860 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:36:11.522826    2557 generic.go:358] "Generic (PLEG): container finished" podID="f02cb33c-c306-4ceb-819f-bd151bd7e3c7" containerID="2be2be0cc6672144eb66adefc155f28370419ec5467577a6349f0d671031baef" exitCode=0
May 25 13:36:11.523335 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:36:11.522887    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-5798769bxrd2g" event={"ID":"f02cb33c-c306-4ceb-819f-bd151bd7e3c7","Type":"ContainerDied","Data":"2be2be0cc6672144eb66adefc155f28370419ec5467577a6349f0d671031baef"}
May 25 13:36:11.523335 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:36:11.522917    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-5798769bxrd2g" event={"ID":"f02cb33c-c306-4ceb-819f-bd151bd7e3c7","Type":"ContainerDied","Data":"1a88d4ee0b171d39e3ef71b2bb5000d6a5e89ba871d87b16480525f450dc0486"}
May 25 13:36:11.523335 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:36:11.522923    2557 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-5798769bxrd2g"
May 25 13:36:11.523335 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:36:11.522932    2557 scope.go:117] "RemoveContainer" containerID="2be2be0cc6672144eb66adefc155f28370419ec5467577a6349f0d671031baef"
May 25 13:36:11.532234 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:36:11.532218    2557 scope.go:117] "RemoveContainer" containerID="82a672b5dd56cb42d84865b8642f117b24160795a8fda712ce124b8de2f7a9d8"
May 25 13:36:11.540281 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:36:11.540259    2557 scope.go:117] "RemoveContainer" containerID="761cef318e370319e1d8f71ddb62e934075db12a6c8c401fdd6c5e7c0daa2ae1"
May 25 13:36:11.548576 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:36:11.548556    2557 scope.go:117] "RemoveContainer" containerID="2be2be0cc6672144eb66adefc155f28370419ec5467577a6349f0d671031baef"
May 25 13:36:11.548893 ip-10-0-143-235 kubenswrapper[2557]: E0525 13:36:11.548874    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2be2be0cc6672144eb66adefc155f28370419ec5467577a6349f0d671031baef\": container with ID starting with 2be2be0cc6672144eb66adefc155f28370419ec5467577a6349f0d671031baef not found: ID does not exist" containerID="2be2be0cc6672144eb66adefc155f28370419ec5467577a6349f0d671031baef"
May 25 13:36:11.548939 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:36:11.548903    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2be2be0cc6672144eb66adefc155f28370419ec5467577a6349f0d671031baef"} err="failed to get container status \"2be2be0cc6672144eb66adefc155f28370419ec5467577a6349f0d671031baef\": rpc error: code = NotFound desc = could not find container \"2be2be0cc6672144eb66adefc155f28370419ec5467577a6349f0d671031baef\": container with ID starting with 2be2be0cc6672144eb66adefc155f28370419ec5467577a6349f0d671031baef not found: ID does not exist"
May 25 13:36:11.548939 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:36:11.548924    2557 scope.go:117] "RemoveContainer" containerID="82a672b5dd56cb42d84865b8642f117b24160795a8fda712ce124b8de2f7a9d8"
May 25 13:36:11.549174 ip-10-0-143-235 kubenswrapper[2557]: E0525 13:36:11.549158    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82a672b5dd56cb42d84865b8642f117b24160795a8fda712ce124b8de2f7a9d8\": container with ID starting with 82a672b5dd56cb42d84865b8642f117b24160795a8fda712ce124b8de2f7a9d8 not found: ID does not exist" containerID="82a672b5dd56cb42d84865b8642f117b24160795a8fda712ce124b8de2f7a9d8"
May 25 13:36:11.549211 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:36:11.549179    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82a672b5dd56cb42d84865b8642f117b24160795a8fda712ce124b8de2f7a9d8"} err="failed to get container status \"82a672b5dd56cb42d84865b8642f117b24160795a8fda712ce124b8de2f7a9d8\": rpc error: code = NotFound desc = could not find container \"82a672b5dd56cb42d84865b8642f117b24160795a8fda712ce124b8de2f7a9d8\": container with ID starting with 82a672b5dd56cb42d84865b8642f117b24160795a8fda712ce124b8de2f7a9d8 not found: ID does not exist"
May 25 13:36:11.549211 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:36:11.549193    2557 scope.go:117] "RemoveContainer" containerID="761cef318e370319e1d8f71ddb62e934075db12a6c8c401fdd6c5e7c0daa2ae1"
May 25 13:36:11.549425 ip-10-0-143-235 kubenswrapper[2557]: E0525 13:36:11.549399    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"761cef318e370319e1d8f71ddb62e934075db12a6c8c401fdd6c5e7c0daa2ae1\": container with ID starting with 761cef318e370319e1d8f71ddb62e934075db12a6c8c401fdd6c5e7c0daa2ae1 not found: ID does not exist" containerID="761cef318e370319e1d8f71ddb62e934075db12a6c8c401fdd6c5e7c0daa2ae1"
May 25 13:36:11.549530 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:36:11.549427    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"761cef318e370319e1d8f71ddb62e934075db12a6c8c401fdd6c5e7c0daa2ae1"} err="failed to get container status \"761cef318e370319e1d8f71ddb62e934075db12a6c8c401fdd6c5e7c0daa2ae1\": rpc error: code = NotFound desc = could not find container \"761cef318e370319e1d8f71ddb62e934075db12a6c8c401fdd6c5e7c0daa2ae1\": container with ID starting with 761cef318e370319e1d8f71ddb62e934075db12a6c8c401fdd6c5e7c0daa2ae1 not found: ID does not exist"
May 25 13:36:11.556469 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:36:11.556435    2557 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-5798769bxrd2g"]
May 25 13:36:11.560044 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:36:11.560019    2557 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-5798769bxrd2g"]
May 25 13:36:12.151103 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:36:12.151073    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f02cb33c-c306-4ceb-819f-bd151bd7e3c7" path="/var/lib/kubelet/pods/f02cb33c-c306-4ceb-819f-bd151bd7e3c7/volumes"
May 25 13:36:20.302617 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:36:20.302540    2557 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-74dcd655dc-5j56f"]
May 25 13:36:20.303136 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:36:20.303117    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f02cb33c-c306-4ceb-819f-bd151bd7e3c7" containerName="tokenizer"
May 25 13:36:20.303239 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:36:20.303139    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="f02cb33c-c306-4ceb-819f-bd151bd7e3c7" containerName="tokenizer"
May 25 13:36:20.303239 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:36:20.303159    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f02cb33c-c306-4ceb-819f-bd151bd7e3c7" containerName="main"
May 25 13:36:20.303239 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:36:20.303167    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="f02cb33c-c306-4ceb-819f-bd151bd7e3c7" containerName="main"
May 25 13:36:20.303239 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:36:20.303194    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f02cb33c-c306-4ceb-819f-bd151bd7e3c7" containerName="storage-initializer"
May 25 13:36:20.303239 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:36:20.303204    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="f02cb33c-c306-4ceb-819f-bd151bd7e3c7" containerName="storage-initializer"
May 25 13:36:20.303530 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:36:20.303291    2557 memory_manager.go:356] "RemoveStaleState removing state" podUID="f02cb33c-c306-4ceb-819f-bd151bd7e3c7" containerName="main"
May 25 13:36:20.303530 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:36:20.303306    2557 memory_manager.go:356] "RemoveStaleState removing state" podUID="f02cb33c-c306-4ceb-819f-bd151bd7e3c7" containerName="tokenizer"
May 25 13:36:20.306637 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:36:20.306614    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-74dcd655dc-5j56f"
May 25 13:36:20.310093 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:36:20.310057    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"router-with-refs-test-kserve-self-signed-certs\""
May 25 13:36:20.311496 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:36:20.311357    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"router-with-refs-test-epp-sa-dockercfg-cptww\""
May 25 13:36:20.311496 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:36:20.311370    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-8blwv\""
May 25 13:36:20.311496 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:36:20.311370    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
May 25 13:36:20.311496 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:36:20.311420    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
May 25 13:36:20.320880 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:36:20.320853    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-74dcd655dc-5j56f"]
May 25 13:36:20.393789 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:36:20.393752    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9wmp\" (UniqueName: \"kubernetes.io/projected/36afa5af-c47e-4135-803d-4e22f1948968-kube-api-access-z9wmp\") pod \"router-with-refs-test-kserve-router-scheduler-74dcd655dc-5j56f\" (UID: \"36afa5af-c47e-4135-803d-4e22f1948968\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-74dcd655dc-5j56f"
May 25 13:36:20.393999 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:36:20.393797    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/36afa5af-c47e-4135-803d-4e22f1948968-tokenizer-uds\") pod \"router-with-refs-test-kserve-router-scheduler-74dcd655dc-5j56f\" (UID: \"36afa5af-c47e-4135-803d-4e22f1948968\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-74dcd655dc-5j56f"
May 25 13:36:20.393999 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:36:20.393870    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/36afa5af-c47e-4135-803d-4e22f1948968-kserve-provision-location\") pod \"router-with-refs-test-kserve-router-scheduler-74dcd655dc-5j56f\" (UID: \"36afa5af-c47e-4135-803d-4e22f1948968\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-74dcd655dc-5j56f"
May 25 13:36:20.393999 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:36:20.393941    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/36afa5af-c47e-4135-803d-4e22f1948968-tls-certs\") pod \"router-with-refs-test-kserve-router-scheduler-74dcd655dc-5j56f\" (UID: \"36afa5af-c47e-4135-803d-4e22f1948968\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-74dcd655dc-5j56f"
May 25 13:36:20.394165 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:36:20.394062    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/36afa5af-c47e-4135-803d-4e22f1948968-tokenizer-tmp\") pod \"router-with-refs-test-kserve-router-scheduler-74dcd655dc-5j56f\" (UID: \"36afa5af-c47e-4135-803d-4e22f1948968\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-74dcd655dc-5j56f"
May 25 13:36:20.394165 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:36:20.394092    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/36afa5af-c47e-4135-803d-4e22f1948968-tokenizer-cache\") pod \"router-with-refs-test-kserve-router-scheduler-74dcd655dc-5j56f\" (UID: \"36afa5af-c47e-4135-803d-4e22f1948968\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-74dcd655dc-5j56f"
May 25 13:36:20.494743 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:36:20.494694    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/36afa5af-c47e-4135-803d-4e22f1948968-tokenizer-tmp\") pod \"router-with-refs-test-kserve-router-scheduler-74dcd655dc-5j56f\" (UID: \"36afa5af-c47e-4135-803d-4e22f1948968\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-74dcd655dc-5j56f"
May 25 13:36:20.494743 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:36:20.494746    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/36afa5af-c47e-4135-803d-4e22f1948968-tokenizer-cache\") pod \"router-with-refs-test-kserve-router-scheduler-74dcd655dc-5j56f\" (UID: \"36afa5af-c47e-4135-803d-4e22f1948968\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-74dcd655dc-5j56f"
May 25 13:36:20.494998 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:36:20.494777    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-z9wmp\" (UniqueName: \"kubernetes.io/projected/36afa5af-c47e-4135-803d-4e22f1948968-kube-api-access-z9wmp\") pod \"router-with-refs-test-kserve-router-scheduler-74dcd655dc-5j56f\" (UID: \"36afa5af-c47e-4135-803d-4e22f1948968\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-74dcd655dc-5j56f"
May 25 13:36:20.494998 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:36:20.494804    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/36afa5af-c47e-4135-803d-4e22f1948968-tokenizer-uds\") pod \"router-with-refs-test-kserve-router-scheduler-74dcd655dc-5j56f\" (UID: \"36afa5af-c47e-4135-803d-4e22f1948968\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-74dcd655dc-5j56f"
May 25 13:36:20.494998 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:36:20.494827    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/36afa5af-c47e-4135-803d-4e22f1948968-kserve-provision-location\") pod \"router-with-refs-test-kserve-router-scheduler-74dcd655dc-5j56f\" (UID: \"36afa5af-c47e-4135-803d-4e22f1948968\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-74dcd655dc-5j56f"
May 25 13:36:20.494998 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:36:20.494867    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/36afa5af-c47e-4135-803d-4e22f1948968-tls-certs\") pod \"router-with-refs-test-kserve-router-scheduler-74dcd655dc-5j56f\" (UID: \"36afa5af-c47e-4135-803d-4e22f1948968\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-74dcd655dc-5j56f"
May 25 13:36:20.495198 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:36:20.495122    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/36afa5af-c47e-4135-803d-4e22f1948968-tokenizer-tmp\") pod \"router-with-refs-test-kserve-router-scheduler-74dcd655dc-5j56f\" (UID: \"36afa5af-c47e-4135-803d-4e22f1948968\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-74dcd655dc-5j56f"
May 25 13:36:20.495255 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:36:20.495193    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/36afa5af-c47e-4135-803d-4e22f1948968-tokenizer-cache\") pod \"router-with-refs-test-kserve-router-scheduler-74dcd655dc-5j56f\" (UID: \"36afa5af-c47e-4135-803d-4e22f1948968\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-74dcd655dc-5j56f"
May 25 13:36:20.495255 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:36:20.495228    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/36afa5af-c47e-4135-803d-4e22f1948968-tokenizer-uds\") pod \"router-with-refs-test-kserve-router-scheduler-74dcd655dc-5j56f\" (UID: \"36afa5af-c47e-4135-803d-4e22f1948968\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-74dcd655dc-5j56f"
May 25 13:36:20.495387 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:36:20.495282    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/36afa5af-c47e-4135-803d-4e22f1948968-kserve-provision-location\") pod \"router-with-refs-test-kserve-router-scheduler-74dcd655dc-5j56f\" (UID: \"36afa5af-c47e-4135-803d-4e22f1948968\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-74dcd655dc-5j56f"
May 25 13:36:20.497451 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:36:20.497430    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/36afa5af-c47e-4135-803d-4e22f1948968-tls-certs\") pod \"router-with-refs-test-kserve-router-scheduler-74dcd655dc-5j56f\" (UID: \"36afa5af-c47e-4135-803d-4e22f1948968\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-74dcd655dc-5j56f"
May 25 13:36:20.518066 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:36:20.518026    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9wmp\" (UniqueName: \"kubernetes.io/projected/36afa5af-c47e-4135-803d-4e22f1948968-kube-api-access-z9wmp\") pod \"router-with-refs-test-kserve-router-scheduler-74dcd655dc-5j56f\" (UID: \"36afa5af-c47e-4135-803d-4e22f1948968\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-74dcd655dc-5j56f"
May 25 13:36:20.617699 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:36:20.617608    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-74dcd655dc-5j56f"
May 25 13:36:20.756229 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:36:20.756198    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-74dcd655dc-5j56f"]
May 25 13:36:20.758674 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:36:20.758645    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36afa5af_c47e_4135_803d_4e22f1948968.slice/crio-6ed9da39e79bcdeef6bb2f3501c97c2327ea4e2aa24350dbe45730f51294c9b1 WatchSource:0}: Error finding container 6ed9da39e79bcdeef6bb2f3501c97c2327ea4e2aa24350dbe45730f51294c9b1: Status 404 returned error can't find the container with id 6ed9da39e79bcdeef6bb2f3501c97c2327ea4e2aa24350dbe45730f51294c9b1
May 25 13:36:21.562504 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:36:21.562470    2557 generic.go:358] "Generic (PLEG): container finished" podID="36afa5af-c47e-4135-803d-4e22f1948968" containerID="a83478d4475dfebd0f112eaf1bdea3d1002533d2075d9fbc2714e4510a30628c" exitCode=0
May 25 13:36:21.562934 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:36:21.562536    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-74dcd655dc-5j56f" event={"ID":"36afa5af-c47e-4135-803d-4e22f1948968","Type":"ContainerDied","Data":"a83478d4475dfebd0f112eaf1bdea3d1002533d2075d9fbc2714e4510a30628c"}
May 25 13:36:21.562934 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:36:21.562563    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-74dcd655dc-5j56f" event={"ID":"36afa5af-c47e-4135-803d-4e22f1948968","Type":"ContainerStarted","Data":"6ed9da39e79bcdeef6bb2f3501c97c2327ea4e2aa24350dbe45730f51294c9b1"}
May 25 13:36:22.568218 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:36:22.568178    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-74dcd655dc-5j56f" event={"ID":"36afa5af-c47e-4135-803d-4e22f1948968","Type":"ContainerStarted","Data":"06bbe341e7900e029a7bd5cf72837b8b4f872433703435e26d3f01e05a435a40"}
May 25 13:36:22.568615 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:36:22.568225    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-74dcd655dc-5j56f" event={"ID":"36afa5af-c47e-4135-803d-4e22f1948968","Type":"ContainerStarted","Data":"84fe31b67f9bbfdfe97380fac187102f2277dd725a50f5a0e48ef14c3511a437"}
May 25 13:36:22.568615 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:36:22.568372    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-74dcd655dc-5j56f"
May 25 13:36:22.591427 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:36:22.591329    2557 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-74dcd655dc-5j56f" podStartSLOduration=2.591312199 podStartE2EDuration="2.591312199s" podCreationTimestamp="2026-05-25 13:36:20 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 13:36:22.590303908 +0000 UTC m=+1835.053320100" watchObservedRunningTime="2026-05-25 13:36:22.591312199 +0000 UTC m=+1835.054328353"
May 25 13:36:30.617804 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:36:30.617764    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-74dcd655dc-5j56f"
May 25 13:36:30.618309 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:36:30.617934    2557 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-74dcd655dc-5j56f"
May 25 13:36:30.620538 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:36:30.620511    2557 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-74dcd655dc-5j56f"
May 25 13:36:31.601368 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:36:31.601307    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-74dcd655dc-5j56f"
May 25 13:36:53.609087 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:36:53.609055    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-74dcd655dc-5j56f"
May 25 13:39:45.548958 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:39:45.548918    2557 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-74dcd655dc-5j56f"]
May 25 13:39:45.549524 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:39:45.549274    2557 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-74dcd655dc-5j56f" podUID="36afa5af-c47e-4135-803d-4e22f1948968" containerName="main" containerID="cri-o://84fe31b67f9bbfdfe97380fac187102f2277dd725a50f5a0e48ef14c3511a437" gracePeriod=30
May 25 13:39:45.549524 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:39:45.549360    2557 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-74dcd655dc-5j56f" podUID="36afa5af-c47e-4135-803d-4e22f1948968" containerName="tokenizer" containerID="cri-o://06bbe341e7900e029a7bd5cf72837b8b4f872433703435e26d3f01e05a435a40" gracePeriod=30
May 25 13:39:46.325424 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:39:46.325382    2557 generic.go:358] "Generic (PLEG): container finished" podID="36afa5af-c47e-4135-803d-4e22f1948968" containerID="84fe31b67f9bbfdfe97380fac187102f2277dd725a50f5a0e48ef14c3511a437" exitCode=0
May 25 13:39:46.325616 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:39:46.325446    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-74dcd655dc-5j56f" event={"ID":"36afa5af-c47e-4135-803d-4e22f1948968","Type":"ContainerDied","Data":"84fe31b67f9bbfdfe97380fac187102f2277dd725a50f5a0e48ef14c3511a437"}
May 25 13:39:46.918462 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:39:46.918436    2557 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-74dcd655dc-5j56f"
May 25 13:39:47.008941 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:39:47.008839    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/36afa5af-c47e-4135-803d-4e22f1948968-tls-certs\") pod \"36afa5af-c47e-4135-803d-4e22f1948968\" (UID: \"36afa5af-c47e-4135-803d-4e22f1948968\") "
May 25 13:39:47.008941 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:39:47.008892    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/36afa5af-c47e-4135-803d-4e22f1948968-kserve-provision-location\") pod \"36afa5af-c47e-4135-803d-4e22f1948968\" (UID: \"36afa5af-c47e-4135-803d-4e22f1948968\") "
May 25 13:39:47.008941 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:39:47.008942    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/36afa5af-c47e-4135-803d-4e22f1948968-tokenizer-cache\") pod \"36afa5af-c47e-4135-803d-4e22f1948968\" (UID: \"36afa5af-c47e-4135-803d-4e22f1948968\") "
May 25 13:39:47.009219 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:39:47.008982    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z9wmp\" (UniqueName: \"kubernetes.io/projected/36afa5af-c47e-4135-803d-4e22f1948968-kube-api-access-z9wmp\") pod \"36afa5af-c47e-4135-803d-4e22f1948968\" (UID: \"36afa5af-c47e-4135-803d-4e22f1948968\") "
May 25 13:39:47.009219 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:39:47.009030    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/36afa5af-c47e-4135-803d-4e22f1948968-tokenizer-tmp\") pod \"36afa5af-c47e-4135-803d-4e22f1948968\" (UID: \"36afa5af-c47e-4135-803d-4e22f1948968\") "
May 25 13:39:47.009219 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:39:47.009060    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/36afa5af-c47e-4135-803d-4e22f1948968-tokenizer-uds\") pod \"36afa5af-c47e-4135-803d-4e22f1948968\" (UID: \"36afa5af-c47e-4135-803d-4e22f1948968\") "
May 25 13:39:47.009391 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:39:47.009229    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36afa5af-c47e-4135-803d-4e22f1948968-tokenizer-cache" (OuterVolumeSpecName: "tokenizer-cache") pod "36afa5af-c47e-4135-803d-4e22f1948968" (UID: "36afa5af-c47e-4135-803d-4e22f1948968"). InnerVolumeSpecName "tokenizer-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 13:39:47.009452 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:39:47.009413    2557 reconciler_common.go:299] "Volume detached for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/36afa5af-c47e-4135-803d-4e22f1948968-tokenizer-cache\") on node \"ip-10-0-143-235.ec2.internal\" DevicePath \"\""
May 25 13:39:47.009452 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:39:47.009414    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36afa5af-c47e-4135-803d-4e22f1948968-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "36afa5af-c47e-4135-803d-4e22f1948968" (UID: "36afa5af-c47e-4135-803d-4e22f1948968"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 13:39:47.009521 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:39:47.009442    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36afa5af-c47e-4135-803d-4e22f1948968-tokenizer-tmp" (OuterVolumeSpecName: "tokenizer-tmp") pod "36afa5af-c47e-4135-803d-4e22f1948968" (UID: "36afa5af-c47e-4135-803d-4e22f1948968"). InnerVolumeSpecName "tokenizer-tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 13:39:47.009796 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:39:47.009775    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36afa5af-c47e-4135-803d-4e22f1948968-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "36afa5af-c47e-4135-803d-4e22f1948968" (UID: "36afa5af-c47e-4135-803d-4e22f1948968"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 13:39:47.011049 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:39:47.011021    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36afa5af-c47e-4135-803d-4e22f1948968-kube-api-access-z9wmp" (OuterVolumeSpecName: "kube-api-access-z9wmp") pod "36afa5af-c47e-4135-803d-4e22f1948968" (UID: "36afa5af-c47e-4135-803d-4e22f1948968"). InnerVolumeSpecName "kube-api-access-z9wmp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 13:39:47.011160 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:39:47.011129    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/36afa5af-c47e-4135-803d-4e22f1948968-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "36afa5af-c47e-4135-803d-4e22f1948968" (UID: "36afa5af-c47e-4135-803d-4e22f1948968"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 13:39:47.110844 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:39:47.110801    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z9wmp\" (UniqueName: \"kubernetes.io/projected/36afa5af-c47e-4135-803d-4e22f1948968-kube-api-access-z9wmp\") on node \"ip-10-0-143-235.ec2.internal\" DevicePath \"\""
May 25 13:39:47.110844 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:39:47.110843    2557 reconciler_common.go:299] "Volume detached for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/36afa5af-c47e-4135-803d-4e22f1948968-tokenizer-tmp\") on node \"ip-10-0-143-235.ec2.internal\" DevicePath \"\""
May 25 13:39:47.110844 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:39:47.110856    2557 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/36afa5af-c47e-4135-803d-4e22f1948968-tokenizer-uds\") on node \"ip-10-0-143-235.ec2.internal\" DevicePath \"\""
May 25 13:39:47.111095 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:39:47.110865    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/36afa5af-c47e-4135-803d-4e22f1948968-tls-certs\") on node \"ip-10-0-143-235.ec2.internal\" DevicePath \"\""
May 25 13:39:47.111095 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:39:47.110874    2557 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/36afa5af-c47e-4135-803d-4e22f1948968-kserve-provision-location\") on node \"ip-10-0-143-235.ec2.internal\" DevicePath \"\""
May 25 13:39:47.331207 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:39:47.331171    2557 generic.go:358] "Generic (PLEG): container finished" podID="36afa5af-c47e-4135-803d-4e22f1948968" containerID="06bbe341e7900e029a7bd5cf72837b8b4f872433703435e26d3f01e05a435a40" exitCode=0
May 25 13:39:47.331207 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:39:47.331209    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-74dcd655dc-5j56f" event={"ID":"36afa5af-c47e-4135-803d-4e22f1948968","Type":"ContainerDied","Data":"06bbe341e7900e029a7bd5cf72837b8b4f872433703435e26d3f01e05a435a40"}
May 25 13:39:47.331480 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:39:47.331232    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-74dcd655dc-5j56f" event={"ID":"36afa5af-c47e-4135-803d-4e22f1948968","Type":"ContainerDied","Data":"6ed9da39e79bcdeef6bb2f3501c97c2327ea4e2aa24350dbe45730f51294c9b1"}
May 25 13:39:47.331480 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:39:47.331241    2557 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-74dcd655dc-5j56f"
May 25 13:39:47.331480 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:39:47.331247    2557 scope.go:117] "RemoveContainer" containerID="06bbe341e7900e029a7bd5cf72837b8b4f872433703435e26d3f01e05a435a40"
May 25 13:39:47.340461 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:39:47.340444    2557 scope.go:117] "RemoveContainer" containerID="84fe31b67f9bbfdfe97380fac187102f2277dd725a50f5a0e48ef14c3511a437"
May 25 13:39:47.348299 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:39:47.348278    2557 scope.go:117] "RemoveContainer" containerID="a83478d4475dfebd0f112eaf1bdea3d1002533d2075d9fbc2714e4510a30628c"
May 25 13:39:47.357509 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:39:47.357476    2557 scope.go:117] "RemoveContainer" containerID="06bbe341e7900e029a7bd5cf72837b8b4f872433703435e26d3f01e05a435a40"
May 25 13:39:47.357860 ip-10-0-143-235 kubenswrapper[2557]: E0525 13:39:47.357832    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06bbe341e7900e029a7bd5cf72837b8b4f872433703435e26d3f01e05a435a40\": container with ID starting with 06bbe341e7900e029a7bd5cf72837b8b4f872433703435e26d3f01e05a435a40 not found: ID does not exist" containerID="06bbe341e7900e029a7bd5cf72837b8b4f872433703435e26d3f01e05a435a40"
May 25 13:39:47.357927 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:39:47.357872    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06bbe341e7900e029a7bd5cf72837b8b4f872433703435e26d3f01e05a435a40"} err="failed to get container status \"06bbe341e7900e029a7bd5cf72837b8b4f872433703435e26d3f01e05a435a40\": rpc error: code = NotFound desc = could not find container \"06bbe341e7900e029a7bd5cf72837b8b4f872433703435e26d3f01e05a435a40\": container with ID starting with 06bbe341e7900e029a7bd5cf72837b8b4f872433703435e26d3f01e05a435a40 not found: ID does not exist"
May 25 13:39:47.357927 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:39:47.357896    2557 scope.go:117] "RemoveContainer" containerID="84fe31b67f9bbfdfe97380fac187102f2277dd725a50f5a0e48ef14c3511a437"
May 25 13:39:47.358183 ip-10-0-143-235 kubenswrapper[2557]: E0525 13:39:47.358162    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84fe31b67f9bbfdfe97380fac187102f2277dd725a50f5a0e48ef14c3511a437\": container with ID starting with 84fe31b67f9bbfdfe97380fac187102f2277dd725a50f5a0e48ef14c3511a437 not found: ID does not exist" containerID="84fe31b67f9bbfdfe97380fac187102f2277dd725a50f5a0e48ef14c3511a437"
May 25 13:39:47.358250 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:39:47.358189    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84fe31b67f9bbfdfe97380fac187102f2277dd725a50f5a0e48ef14c3511a437"} err="failed to get container status \"84fe31b67f9bbfdfe97380fac187102f2277dd725a50f5a0e48ef14c3511a437\": rpc error: code = NotFound desc = could not find container \"84fe31b67f9bbfdfe97380fac187102f2277dd725a50f5a0e48ef14c3511a437\": container with ID starting with 84fe31b67f9bbfdfe97380fac187102f2277dd725a50f5a0e48ef14c3511a437 not found: ID does not exist"
May 25 13:39:47.358250 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:39:47.358213    2557 scope.go:117] "RemoveContainer" containerID="a83478d4475dfebd0f112eaf1bdea3d1002533d2075d9fbc2714e4510a30628c"
May 25 13:39:47.358493 ip-10-0-143-235 kubenswrapper[2557]: E0525 13:39:47.358472    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a83478d4475dfebd0f112eaf1bdea3d1002533d2075d9fbc2714e4510a30628c\": container with ID starting with a83478d4475dfebd0f112eaf1bdea3d1002533d2075d9fbc2714e4510a30628c not found: ID does not exist" containerID="a83478d4475dfebd0f112eaf1bdea3d1002533d2075d9fbc2714e4510a30628c"
May 25 13:39:47.358560 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:39:47.358497    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a83478d4475dfebd0f112eaf1bdea3d1002533d2075d9fbc2714e4510a30628c"} err="failed to get container status \"a83478d4475dfebd0f112eaf1bdea3d1002533d2075d9fbc2714e4510a30628c\": rpc error: code = NotFound desc = could not find container \"a83478d4475dfebd0f112eaf1bdea3d1002533d2075d9fbc2714e4510a30628c\": container with ID starting with a83478d4475dfebd0f112eaf1bdea3d1002533d2075d9fbc2714e4510a30628c not found: ID does not exist"
May 25 13:39:47.366409 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:39:47.366307    2557 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-74dcd655dc-5j56f"]
May 25 13:39:47.369133 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:39:47.369108    2557 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-74dcd655dc-5j56f"]
May 25 13:39:48.151615 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:39:48.151583    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="36afa5af-c47e-4135-803d-4e22f1948968" path="/var/lib/kubelet/pods/36afa5af-c47e-4135-803d-4e22f1948968/volumes"
May 25 13:39:56.083320 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:39:56.083278    2557 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schesdn9c"]
May 25 13:39:56.083929 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:39:56.083907    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="36afa5af-c47e-4135-803d-4e22f1948968" containerName="main"
May 25 13:39:56.083977 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:39:56.083934    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="36afa5af-c47e-4135-803d-4e22f1948968" containerName="main"
May 25 13:39:56.083977 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:39:56.083951    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="36afa5af-c47e-4135-803d-4e22f1948968" containerName="tokenizer"
May 25 13:39:56.083977 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:39:56.083960    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="36afa5af-c47e-4135-803d-4e22f1948968" containerName="tokenizer"
May 25 13:39:56.084066 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:39:56.083993    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="36afa5af-c47e-4135-803d-4e22f1948968" containerName="storage-initializer"
May 25 13:39:56.084066 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:39:56.084003    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="36afa5af-c47e-4135-803d-4e22f1948968" containerName="storage-initializer"
May 25 13:39:56.084132 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:39:56.084098    2557 memory_manager.go:356] "RemoveStaleState removing state" podUID="36afa5af-c47e-4135-803d-4e22f1948968" containerName="tokenizer"
May 25 13:39:56.084132 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:39:56.084111    2557 memory_manager.go:356] "RemoveStaleState removing state" podUID="36afa5af-c47e-4135-803d-4e22f1948968" containerName="main"
May 25 13:39:56.087767 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:39:56.087742    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schesdn9c"
May 25 13:39:56.090739 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:39:56.090710    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-8blwv\""
May 25 13:39:56.090876 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:39:56.090744    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
May 25 13:39:56.090876 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:39:56.090709    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-fb-opt-125m-route-05aa9bba-epp-sa-dockercfg-mwzhl\""
May 25 13:39:56.091740 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:39:56.091722    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
May 25 13:39:56.091896 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:39:56.091878    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv8f1a6f044e8c7a4d31a250e0c4861caf-kserve-self-signed-certs\""
May 25 13:39:56.098901 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:39:56.098870    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schesdn9c"]
May 25 13:39:56.194128 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:39:56.194091    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/98e798c2-e747-4978-b9a4-bd095e49d914-tls-certs\") pod \"llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schesdn9c\" (UID: \"98e798c2-e747-4978-b9a4-bd095e49d914\") " pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schesdn9c"
May 25 13:39:56.194314 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:39:56.194152    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jbql\" (UniqueName: \"kubernetes.io/projected/98e798c2-e747-4978-b9a4-bd095e49d914-kube-api-access-5jbql\") pod \"llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schesdn9c\" (UID: \"98e798c2-e747-4978-b9a4-bd095e49d914\") " pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schesdn9c"
May 25 13:39:56.194314 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:39:56.194183    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/98e798c2-e747-4978-b9a4-bd095e49d914-tokenizer-uds\") pod \"llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schesdn9c\" (UID: \"98e798c2-e747-4978-b9a4-bd095e49d914\") " pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schesdn9c"
May 25 13:39:56.194314 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:39:56.194206    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/98e798c2-e747-4978-b9a4-bd095e49d914-tokenizer-cache\") pod \"llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schesdn9c\" (UID: \"98e798c2-e747-4978-b9a4-bd095e49d914\") " pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schesdn9c"
May 25 13:39:56.194314 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:39:56.194231    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/98e798c2-e747-4978-b9a4-bd095e49d914-tokenizer-tmp\") pod \"llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schesdn9c\" (UID: \"98e798c2-e747-4978-b9a4-bd095e49d914\") " pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schesdn9c"
May 25 13:39:56.194509 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:39:56.194311    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/98e798c2-e747-4978-b9a4-bd095e49d914-kserve-provision-location\") pod \"llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schesdn9c\" (UID: \"98e798c2-e747-4978-b9a4-bd095e49d914\") " pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schesdn9c"
May 25 13:39:56.295783 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:39:56.295727    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-5jbql\" (UniqueName: \"kubernetes.io/projected/98e798c2-e747-4978-b9a4-bd095e49d914-kube-api-access-5jbql\") pod \"llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schesdn9c\" (UID: \"98e798c2-e747-4978-b9a4-bd095e49d914\") " pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schesdn9c"
May 25 13:39:56.295993 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:39:56.295795    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/98e798c2-e747-4978-b9a4-bd095e49d914-tokenizer-uds\") pod \"llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schesdn9c\" (UID: \"98e798c2-e747-4978-b9a4-bd095e49d914\") " pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schesdn9c"
May 25 13:39:56.295993 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:39:56.295823    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/98e798c2-e747-4978-b9a4-bd095e49d914-tokenizer-cache\") pod \"llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schesdn9c\" (UID: \"98e798c2-e747-4978-b9a4-bd095e49d914\") " pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schesdn9c"
May 25 13:39:56.295993 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:39:56.295849    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/98e798c2-e747-4978-b9a4-bd095e49d914-tokenizer-tmp\") pod \"llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schesdn9c\" (UID: \"98e798c2-e747-4978-b9a4-bd095e49d914\") " pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schesdn9c"
May 25 13:39:56.296146 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:39:56.296009    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/98e798c2-e747-4978-b9a4-bd095e49d914-kserve-provision-location\") pod \"llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schesdn9c\" (UID: \"98e798c2-e747-4978-b9a4-bd095e49d914\") " pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schesdn9c"
May 25 13:39:56.296146 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:39:56.296115    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/98e798c2-e747-4978-b9a4-bd095e49d914-tls-certs\") pod \"llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schesdn9c\" (UID: \"98e798c2-e747-4978-b9a4-bd095e49d914\") " pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schesdn9c"
May 25 13:39:56.296273 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:39:56.296251    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/98e798c2-e747-4978-b9a4-bd095e49d914-tokenizer-uds\") pod \"llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schesdn9c\" (UID: \"98e798c2-e747-4978-b9a4-bd095e49d914\") " pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schesdn9c"
May 25 13:39:56.296328 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:39:56.296293    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/98e798c2-e747-4978-b9a4-bd095e49d914-tokenizer-cache\") pod \"llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schesdn9c\" (UID: \"98e798c2-e747-4978-b9a4-bd095e49d914\") " pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schesdn9c"
May 25 13:39:56.296418 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:39:56.296331    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/98e798c2-e747-4978-b9a4-bd095e49d914-tokenizer-tmp\") pod \"llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schesdn9c\" (UID: \"98e798c2-e747-4978-b9a4-bd095e49d914\") " pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schesdn9c"
May 25 13:39:56.296418 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:39:56.296402    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/98e798c2-e747-4978-b9a4-bd095e49d914-kserve-provision-location\") pod \"llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schesdn9c\" (UID: \"98e798c2-e747-4978-b9a4-bd095e49d914\") " pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schesdn9c"
May 25 13:39:56.298596 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:39:56.298570    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/98e798c2-e747-4978-b9a4-bd095e49d914-tls-certs\") pod \"llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schesdn9c\" (UID: \"98e798c2-e747-4978-b9a4-bd095e49d914\") " pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schesdn9c"
May 25 13:39:56.304331 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:39:56.304301    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jbql\" (UniqueName: \"kubernetes.io/projected/98e798c2-e747-4978-b9a4-bd095e49d914-kube-api-access-5jbql\") pod \"llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schesdn9c\" (UID: \"98e798c2-e747-4978-b9a4-bd095e49d914\") " pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schesdn9c"
May 25 13:39:56.400303 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:39:56.400199    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schesdn9c"
May 25 13:39:56.544109 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:39:56.544077    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schesdn9c"]
May 25 13:39:56.546215 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:39:56.546184    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98e798c2_e747_4978_b9a4_bd095e49d914.slice/crio-4eb4b9cf150e70426bac645e745983268cd19df5307040a7bbbd2bfd936aeb92 WatchSource:0}: Error finding container 4eb4b9cf150e70426bac645e745983268cd19df5307040a7bbbd2bfd936aeb92: Status 404 returned error can't find the container with id 4eb4b9cf150e70426bac645e745983268cd19df5307040a7bbbd2bfd936aeb92
May 25 13:39:56.548055 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:39:56.548036    2557 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 25 13:39:57.368775 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:39:57.368732    2557 generic.go:358] "Generic (PLEG): container finished" podID="98e798c2-e747-4978-b9a4-bd095e49d914" containerID="133baf55a644109688241fb91413a1e132e1ae5bf3fe1a0e57343d40534aaac4" exitCode=0
May 25 13:39:57.369125 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:39:57.368802    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schesdn9c" event={"ID":"98e798c2-e747-4978-b9a4-bd095e49d914","Type":"ContainerDied","Data":"133baf55a644109688241fb91413a1e132e1ae5bf3fe1a0e57343d40534aaac4"}
May 25 13:39:57.369125 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:39:57.368840    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schesdn9c" event={"ID":"98e798c2-e747-4978-b9a4-bd095e49d914","Type":"ContainerStarted","Data":"4eb4b9cf150e70426bac645e745983268cd19df5307040a7bbbd2bfd936aeb92"}
May 25 13:39:58.374440 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:39:58.374395    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schesdn9c" event={"ID":"98e798c2-e747-4978-b9a4-bd095e49d914","Type":"ContainerStarted","Data":"9f3928c2a4db8400c080fe7adda8ed59ac9b088b9490db71766951c5dd522741"}
May 25 13:39:58.374858 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:39:58.374449    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schesdn9c" event={"ID":"98e798c2-e747-4978-b9a4-bd095e49d914","Type":"ContainerStarted","Data":"643541b43ea957e29961fe9ef068b583d99d5327251b9004d3fde71852ae69ea"}
May 25 13:39:58.374858 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:39:58.374609    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schesdn9c"
May 25 13:39:58.398548 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:39:58.398497    2557 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schesdn9c" podStartSLOduration=2.398478971 podStartE2EDuration="2.398478971s" podCreationTimestamp="2026-05-25 13:39:56 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 13:39:58.396190969 +0000 UTC m=+2050.859207127" watchObservedRunningTime="2026-05-25 13:39:58.398478971 +0000 UTC m=+2050.861495127"
May 25 13:40:06.400992 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:40:06.400950    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schesdn9c"
May 25 13:40:06.400992 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:40:06.400987    2557 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schesdn9c"
May 25 13:40:06.403614 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:40:06.403582    2557 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schesdn9c"
May 25 13:40:06.407240 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:40:06.407212    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schesdn9c"
May 25 13:40:27.414197 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:40:27.414109    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schesdn9c"
May 25 13:40:48.205141 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:40:48.205105    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vqnvc_d21bdf8b-d0ab-48f2-8736-54e92947dd53/ovn-acl-logging/0.log"
May 25 13:40:48.211716 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:40:48.211691    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vqnvc_d21bdf8b-d0ab-48f2-8736-54e92947dd53/ovn-acl-logging/0.log"
May 25 13:43:44.122285 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:43:44.122247    2557 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schesdn9c"]
May 25 13:43:44.122829 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:43:44.122587    2557 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schesdn9c" podUID="98e798c2-e747-4978-b9a4-bd095e49d914" containerName="main" containerID="cri-o://643541b43ea957e29961fe9ef068b583d99d5327251b9004d3fde71852ae69ea" gracePeriod=30
May 25 13:43:44.122829 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:43:44.122637    2557 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schesdn9c" podUID="98e798c2-e747-4978-b9a4-bd095e49d914" containerName="tokenizer" containerID="cri-o://9f3928c2a4db8400c080fe7adda8ed59ac9b088b9490db71766951c5dd522741" gracePeriod=30
May 25 13:43:45.215273 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:43:45.215230    2557 generic.go:358] "Generic (PLEG): container finished" podID="98e798c2-e747-4978-b9a4-bd095e49d914" containerID="643541b43ea957e29961fe9ef068b583d99d5327251b9004d3fde71852ae69ea" exitCode=0
May 25 13:43:45.215653 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:43:45.215303    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schesdn9c" event={"ID":"98e798c2-e747-4978-b9a4-bd095e49d914","Type":"ContainerDied","Data":"643541b43ea957e29961fe9ef068b583d99d5327251b9004d3fde71852ae69ea"}
May 25 13:43:45.581484 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:43:45.581459    2557 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schesdn9c"
May 25 13:43:45.618762 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:43:45.618731    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/98e798c2-e747-4978-b9a4-bd095e49d914-tokenizer-uds\") pod \"98e798c2-e747-4978-b9a4-bd095e49d914\" (UID: \"98e798c2-e747-4978-b9a4-bd095e49d914\") "
May 25 13:43:45.618762 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:43:45.618775    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/98e798c2-e747-4978-b9a4-bd095e49d914-tokenizer-cache\") pod \"98e798c2-e747-4978-b9a4-bd095e49d914\" (UID: \"98e798c2-e747-4978-b9a4-bd095e49d914\") "
May 25 13:43:45.619003 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:43:45.618853    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/98e798c2-e747-4978-b9a4-bd095e49d914-kserve-provision-location\") pod \"98e798c2-e747-4978-b9a4-bd095e49d914\" (UID: \"98e798c2-e747-4978-b9a4-bd095e49d914\") "
May 25 13:43:45.619003 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:43:45.618898    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5jbql\" (UniqueName: \"kubernetes.io/projected/98e798c2-e747-4978-b9a4-bd095e49d914-kube-api-access-5jbql\") pod \"98e798c2-e747-4978-b9a4-bd095e49d914\" (UID: \"98e798c2-e747-4978-b9a4-bd095e49d914\") "
May 25 13:43:45.619003 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:43:45.618926    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/98e798c2-e747-4978-b9a4-bd095e49d914-tokenizer-tmp\") pod \"98e798c2-e747-4978-b9a4-bd095e49d914\" (UID: \"98e798c2-e747-4978-b9a4-bd095e49d914\") "
May 25 13:43:45.619003 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:43:45.618953    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/98e798c2-e747-4978-b9a4-bd095e49d914-tls-certs\") pod \"98e798c2-e747-4978-b9a4-bd095e49d914\" (UID: \"98e798c2-e747-4978-b9a4-bd095e49d914\") "
May 25 13:43:45.619208 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:43:45.619054    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98e798c2-e747-4978-b9a4-bd095e49d914-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "98e798c2-e747-4978-b9a4-bd095e49d914" (UID: "98e798c2-e747-4978-b9a4-bd095e49d914"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 13:43:45.619208 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:43:45.619096    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98e798c2-e747-4978-b9a4-bd095e49d914-tokenizer-cache" (OuterVolumeSpecName: "tokenizer-cache") pod "98e798c2-e747-4978-b9a4-bd095e49d914" (UID: "98e798c2-e747-4978-b9a4-bd095e49d914"). InnerVolumeSpecName "tokenizer-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 13:43:45.619308 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:43:45.619287    2557 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/98e798c2-e747-4978-b9a4-bd095e49d914-tokenizer-uds\") on node \"ip-10-0-143-235.ec2.internal\" DevicePath \"\""
May 25 13:43:45.619308 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:43:45.619285    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98e798c2-e747-4978-b9a4-bd095e49d914-tokenizer-tmp" (OuterVolumeSpecName: "tokenizer-tmp") pod "98e798c2-e747-4978-b9a4-bd095e49d914" (UID: "98e798c2-e747-4978-b9a4-bd095e49d914"). InnerVolumeSpecName "tokenizer-tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 13:43:45.619435 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:43:45.619307    2557 reconciler_common.go:299] "Volume detached for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/98e798c2-e747-4978-b9a4-bd095e49d914-tokenizer-cache\") on node \"ip-10-0-143-235.ec2.internal\" DevicePath \"\""
May 25 13:43:45.619688 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:43:45.619664    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98e798c2-e747-4978-b9a4-bd095e49d914-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "98e798c2-e747-4978-b9a4-bd095e49d914" (UID: "98e798c2-e747-4978-b9a4-bd095e49d914"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 13:43:45.621044 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:43:45.621017    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98e798c2-e747-4978-b9a4-bd095e49d914-kube-api-access-5jbql" (OuterVolumeSpecName: "kube-api-access-5jbql") pod "98e798c2-e747-4978-b9a4-bd095e49d914" (UID: "98e798c2-e747-4978-b9a4-bd095e49d914"). InnerVolumeSpecName "kube-api-access-5jbql". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 13:43:45.621044 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:43:45.621028    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/98e798c2-e747-4978-b9a4-bd095e49d914-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "98e798c2-e747-4978-b9a4-bd095e49d914" (UID: "98e798c2-e747-4978-b9a4-bd095e49d914"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 13:43:45.720616 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:43:45.720582    2557 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/98e798c2-e747-4978-b9a4-bd095e49d914-kserve-provision-location\") on node \"ip-10-0-143-235.ec2.internal\" DevicePath \"\""
May 25 13:43:45.720616 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:43:45.720613    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5jbql\" (UniqueName: \"kubernetes.io/projected/98e798c2-e747-4978-b9a4-bd095e49d914-kube-api-access-5jbql\") on node \"ip-10-0-143-235.ec2.internal\" DevicePath \"\""
May 25 13:43:45.720616 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:43:45.720626    2557 reconciler_common.go:299] "Volume detached for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/98e798c2-e747-4978-b9a4-bd095e49d914-tokenizer-tmp\") on node \"ip-10-0-143-235.ec2.internal\" DevicePath \"\""
May 25 13:43:45.720852 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:43:45.720641    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/98e798c2-e747-4978-b9a4-bd095e49d914-tls-certs\") on node \"ip-10-0-143-235.ec2.internal\" DevicePath \"\""
May 25 13:43:46.220307 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:43:46.220268    2557 generic.go:358] "Generic (PLEG): container finished" podID="98e798c2-e747-4978-b9a4-bd095e49d914" containerID="9f3928c2a4db8400c080fe7adda8ed59ac9b088b9490db71766951c5dd522741" exitCode=0
May 25 13:43:46.220701 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:43:46.220337    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schesdn9c" event={"ID":"98e798c2-e747-4978-b9a4-bd095e49d914","Type":"ContainerDied","Data":"9f3928c2a4db8400c080fe7adda8ed59ac9b088b9490db71766951c5dd522741"}
May 25 13:43:46.220701 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:43:46.220366    2557 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schesdn9c"
May 25 13:43:46.220701 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:43:46.220389    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schesdn9c" event={"ID":"98e798c2-e747-4978-b9a4-bd095e49d914","Type":"ContainerDied","Data":"4eb4b9cf150e70426bac645e745983268cd19df5307040a7bbbd2bfd936aeb92"}
May 25 13:43:46.220701 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:43:46.220410    2557 scope.go:117] "RemoveContainer" containerID="9f3928c2a4db8400c080fe7adda8ed59ac9b088b9490db71766951c5dd522741"
May 25 13:43:46.228627 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:43:46.228602    2557 scope.go:117] "RemoveContainer" containerID="643541b43ea957e29961fe9ef068b583d99d5327251b9004d3fde71852ae69ea"
May 25 13:43:46.236243 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:43:46.236222    2557 scope.go:117] "RemoveContainer" containerID="133baf55a644109688241fb91413a1e132e1ae5bf3fe1a0e57343d40534aaac4"
May 25 13:43:46.240056 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:43:46.240033    2557 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schesdn9c"]
May 25 13:43:46.244866 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:43:46.244845    2557 scope.go:117] "RemoveContainer" containerID="9f3928c2a4db8400c080fe7adda8ed59ac9b088b9490db71766951c5dd522741"
May 25 13:43:46.245174 ip-10-0-143-235 kubenswrapper[2557]: E0525 13:43:46.245152    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f3928c2a4db8400c080fe7adda8ed59ac9b088b9490db71766951c5dd522741\": container with ID starting with 9f3928c2a4db8400c080fe7adda8ed59ac9b088b9490db71766951c5dd522741 not found: ID does not exist" containerID="9f3928c2a4db8400c080fe7adda8ed59ac9b088b9490db71766951c5dd522741"
May 25 13:43:46.245287 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:43:46.245183    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f3928c2a4db8400c080fe7adda8ed59ac9b088b9490db71766951c5dd522741"} err="failed to get container status \"9f3928c2a4db8400c080fe7adda8ed59ac9b088b9490db71766951c5dd522741\": rpc error: code = NotFound desc = could not find container \"9f3928c2a4db8400c080fe7adda8ed59ac9b088b9490db71766951c5dd522741\": container with ID starting with 9f3928c2a4db8400c080fe7adda8ed59ac9b088b9490db71766951c5dd522741 not found: ID does not exist"
May 25 13:43:46.245287 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:43:46.245202    2557 scope.go:117] "RemoveContainer" containerID="643541b43ea957e29961fe9ef068b583d99d5327251b9004d3fde71852ae69ea"
May 25 13:43:46.245508 ip-10-0-143-235 kubenswrapper[2557]: E0525 13:43:46.245481    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"643541b43ea957e29961fe9ef068b583d99d5327251b9004d3fde71852ae69ea\": container with ID starting with 643541b43ea957e29961fe9ef068b583d99d5327251b9004d3fde71852ae69ea not found: ID does not exist" containerID="643541b43ea957e29961fe9ef068b583d99d5327251b9004d3fde71852ae69ea"
May 25 13:43:46.245583 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:43:46.245508    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"643541b43ea957e29961fe9ef068b583d99d5327251b9004d3fde71852ae69ea"} err="failed to get container status \"643541b43ea957e29961fe9ef068b583d99d5327251b9004d3fde71852ae69ea\": rpc error: code = NotFound desc = could not find container \"643541b43ea957e29961fe9ef068b583d99d5327251b9004d3fde71852ae69ea\": container with ID starting with 643541b43ea957e29961fe9ef068b583d99d5327251b9004d3fde71852ae69ea not found: ID does not exist"
May 25 13:43:46.245583 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:43:46.245522    2557 scope.go:117] "RemoveContainer" containerID="133baf55a644109688241fb91413a1e132e1ae5bf3fe1a0e57343d40534aaac4"
May 25 13:43:46.245796 ip-10-0-143-235 kubenswrapper[2557]: E0525 13:43:46.245781    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"133baf55a644109688241fb91413a1e132e1ae5bf3fe1a0e57343d40534aaac4\": container with ID starting with 133baf55a644109688241fb91413a1e132e1ae5bf3fe1a0e57343d40534aaac4 not found: ID does not exist" containerID="133baf55a644109688241fb91413a1e132e1ae5bf3fe1a0e57343d40534aaac4"
May 25 13:43:46.245850 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:43:46.245798    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"133baf55a644109688241fb91413a1e132e1ae5bf3fe1a0e57343d40534aaac4"} err="failed to get container status \"133baf55a644109688241fb91413a1e132e1ae5bf3fe1a0e57343d40534aaac4\": rpc error: code = NotFound desc = could not find container \"133baf55a644109688241fb91413a1e132e1ae5bf3fe1a0e57343d40534aaac4\": container with ID starting with 133baf55a644109688241fb91413a1e132e1ae5bf3fe1a0e57343d40534aaac4 not found: ID does not exist"
May 25 13:43:46.245904 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:43:46.245850    2557 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schesdn9c"]
May 25 13:43:48.151534 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:43:48.151498    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="98e798c2-e747-4978-b9a4-bd095e49d914" path="/var/lib/kubelet/pods/98e798c2-e747-4978-b9a4-bd095e49d914/volumes"
May 25 13:43:48.880059 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:43:48.880020    2557 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-86bd7ccwkq"]
May 25 13:43:48.880614 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:43:48.880598    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="98e798c2-e747-4978-b9a4-bd095e49d914" containerName="tokenizer"
May 25 13:43:48.880663 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:43:48.880618    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="98e798c2-e747-4978-b9a4-bd095e49d914" containerName="tokenizer"
May 25 13:43:48.880663 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:43:48.880643    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="98e798c2-e747-4978-b9a4-bd095e49d914" containerName="main"
May 25 13:43:48.880663 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:43:48.880652    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="98e798c2-e747-4978-b9a4-bd095e49d914" containerName="main"
May 25 13:43:48.880752 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:43:48.880684    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="98e798c2-e747-4978-b9a4-bd095e49d914" containerName="storage-initializer"
May 25 13:43:48.880752 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:43:48.880693    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="98e798c2-e747-4978-b9a4-bd095e49d914" containerName="storage-initializer"
May 25 13:43:48.880814 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:43:48.880775    2557 memory_manager.go:356] "RemoveStaleState removing state" podUID="98e798c2-e747-4978-b9a4-bd095e49d914" containerName="tokenizer"
May 25 13:43:48.880814 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:43:48.880790    2557 memory_manager.go:356] "RemoveStaleState removing state" podUID="98e798c2-e747-4978-b9a4-bd095e49d914" containerName="main"
May 25 13:43:48.886212 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:43:48.886188    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-86bd7ccwkq"
May 25 13:43:48.889674 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:43:48.889653    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
May 25 13:43:48.889789 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:43:48.889655    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-8blwv\""
May 25 13:43:48.890828 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:43:48.890807    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"custom-route-timeout-pd-test-kserve-self-signed-certs\""
May 25 13:43:48.890976 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:43:48.890830    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"custom-route-timeout-pd-test-epp-sa-dockercfg-bj2tt\""
May 25 13:43:48.890976 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:43:48.890846    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
May 25 13:43:48.898026 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:43:48.898003    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-86bd7ccwkq"]
May 25 13:43:48.950380 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:43:48.950306    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b29dde20-ae27-43e1-80ae-de916e45fc1e-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-86bd7ccwkq\" (UID: \"b29dde20-ae27-43e1-80ae-de916e45fc1e\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-86bd7ccwkq"
May 25 13:43:48.950581 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:43:48.950463    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b29dde20-ae27-43e1-80ae-de916e45fc1e-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-86bd7ccwkq\" (UID: \"b29dde20-ae27-43e1-80ae-de916e45fc1e\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-86bd7ccwkq"
May 25 13:43:48.950581 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:43:48.950503    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/b29dde20-ae27-43e1-80ae-de916e45fc1e-tokenizer-uds\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-86bd7ccwkq\" (UID: \"b29dde20-ae27-43e1-80ae-de916e45fc1e\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-86bd7ccwkq"
May 25 13:43:48.950581 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:43:48.950533    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/b29dde20-ae27-43e1-80ae-de916e45fc1e-tokenizer-tmp\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-86bd7ccwkq\" (UID: \"b29dde20-ae27-43e1-80ae-de916e45fc1e\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-86bd7ccwkq"
May 25 13:43:48.950701 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:43:48.950600    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rccg\" (UniqueName: \"kubernetes.io/projected/b29dde20-ae27-43e1-80ae-de916e45fc1e-kube-api-access-7rccg\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-86bd7ccwkq\" (UID: \"b29dde20-ae27-43e1-80ae-de916e45fc1e\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-86bd7ccwkq"
May 25 13:43:48.950701 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:43:48.950639    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/b29dde20-ae27-43e1-80ae-de916e45fc1e-tokenizer-cache\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-86bd7ccwkq\" (UID: \"b29dde20-ae27-43e1-80ae-de916e45fc1e\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-86bd7ccwkq"
May 25 13:43:49.051953 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:43:49.051910    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b29dde20-ae27-43e1-80ae-de916e45fc1e-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-86bd7ccwkq\" (UID: \"b29dde20-ae27-43e1-80ae-de916e45fc1e\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-86bd7ccwkq"
May 25 13:43:49.051953 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:43:49.051954    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/b29dde20-ae27-43e1-80ae-de916e45fc1e-tokenizer-uds\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-86bd7ccwkq\" (UID: \"b29dde20-ae27-43e1-80ae-de916e45fc1e\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-86bd7ccwkq"
May 25 13:43:49.052211 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:43:49.052082    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/b29dde20-ae27-43e1-80ae-de916e45fc1e-tokenizer-tmp\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-86bd7ccwkq\" (UID: \"b29dde20-ae27-43e1-80ae-de916e45fc1e\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-86bd7ccwkq"
May 25 13:43:49.052211 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:43:49.052146    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-7rccg\" (UniqueName: \"kubernetes.io/projected/b29dde20-ae27-43e1-80ae-de916e45fc1e-kube-api-access-7rccg\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-86bd7ccwkq\" (UID: \"b29dde20-ae27-43e1-80ae-de916e45fc1e\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-86bd7ccwkq"
May 25 13:43:49.052211 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:43:49.052172    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/b29dde20-ae27-43e1-80ae-de916e45fc1e-tokenizer-cache\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-86bd7ccwkq\" (UID: \"b29dde20-ae27-43e1-80ae-de916e45fc1e\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-86bd7ccwkq"
May 25 13:43:49.052587 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:43:49.052215    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b29dde20-ae27-43e1-80ae-de916e45fc1e-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-86bd7ccwkq\" (UID: \"b29dde20-ae27-43e1-80ae-de916e45fc1e\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-86bd7ccwkq"
May 25 13:43:49.052587 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:43:49.052390    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/b29dde20-ae27-43e1-80ae-de916e45fc1e-tokenizer-uds\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-86bd7ccwkq\" (UID: \"b29dde20-ae27-43e1-80ae-de916e45fc1e\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-86bd7ccwkq"
May 25 13:43:49.052587 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:43:49.052461    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b29dde20-ae27-43e1-80ae-de916e45fc1e-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-86bd7ccwkq\" (UID: \"b29dde20-ae27-43e1-80ae-de916e45fc1e\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-86bd7ccwkq"
May 25 13:43:49.052587 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:43:49.052471    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/b29dde20-ae27-43e1-80ae-de916e45fc1e-tokenizer-tmp\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-86bd7ccwkq\" (UID: \"b29dde20-ae27-43e1-80ae-de916e45fc1e\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-86bd7ccwkq"
May 25 13:43:49.052587 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:43:49.052576    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/b29dde20-ae27-43e1-80ae-de916e45fc1e-tokenizer-cache\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-86bd7ccwkq\" (UID: \"b29dde20-ae27-43e1-80ae-de916e45fc1e\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-86bd7ccwkq"
May 25 13:43:49.054839 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:43:49.054817    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b29dde20-ae27-43e1-80ae-de916e45fc1e-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-86bd7ccwkq\" (UID: \"b29dde20-ae27-43e1-80ae-de916e45fc1e\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-86bd7ccwkq"
May 25 13:43:49.061492 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:43:49.061466    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rccg\" (UniqueName: \"kubernetes.io/projected/b29dde20-ae27-43e1-80ae-de916e45fc1e-kube-api-access-7rccg\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-86bd7ccwkq\" (UID: \"b29dde20-ae27-43e1-80ae-de916e45fc1e\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-86bd7ccwkq"
May 25 13:43:49.196019 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:43:49.195913    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-86bd7ccwkq"
May 25 13:43:49.338310 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:43:49.338282    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-86bd7ccwkq"]
May 25 13:43:49.341202 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:43:49.341176    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb29dde20_ae27_43e1_80ae_de916e45fc1e.slice/crio-0d0234ee5f526b060214b20baf8ee01a772b64c3814db112e282d812a474c94f WatchSource:0}: Error finding container 0d0234ee5f526b060214b20baf8ee01a772b64c3814db112e282d812a474c94f: Status 404 returned error can't find the container with id 0d0234ee5f526b060214b20baf8ee01a772b64c3814db112e282d812a474c94f
May 25 13:43:50.239437 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:43:50.239399    2557 generic.go:358] "Generic (PLEG): container finished" podID="b29dde20-ae27-43e1-80ae-de916e45fc1e" containerID="7eea61e441b0d8fe44aa1726b7039d413511406bd03173aa939b725f645010e8" exitCode=0
May 25 13:43:50.239843 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:43:50.239484    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-86bd7ccwkq" event={"ID":"b29dde20-ae27-43e1-80ae-de916e45fc1e","Type":"ContainerDied","Data":"7eea61e441b0d8fe44aa1726b7039d413511406bd03173aa939b725f645010e8"}
May 25 13:43:50.239843 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:43:50.239513    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-86bd7ccwkq" event={"ID":"b29dde20-ae27-43e1-80ae-de916e45fc1e","Type":"ContainerStarted","Data":"0d0234ee5f526b060214b20baf8ee01a772b64c3814db112e282d812a474c94f"}
May 25 13:43:51.246202 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:43:51.246166    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-86bd7ccwkq" event={"ID":"b29dde20-ae27-43e1-80ae-de916e45fc1e","Type":"ContainerStarted","Data":"12dbe62cbc7580b292a41d3dffeda340f1526b70f9a39d62d37179297975ae10"}
May 25 13:43:51.246202 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:43:51.246208    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-86bd7ccwkq" event={"ID":"b29dde20-ae27-43e1-80ae-de916e45fc1e","Type":"ContainerStarted","Data":"e6e54ceb9cf7e143f61f3b2c25e0f12d9b3b0bbfd06fddbd84a99c3d89081b11"}
May 25 13:43:51.246658 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:43:51.246246    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-86bd7ccwkq"
May 25 13:43:51.272006 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:43:51.271941    2557 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-86bd7ccwkq" podStartSLOduration=3.271920726 podStartE2EDuration="3.271920726s" podCreationTimestamp="2026-05-25 13:43:48 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 13:43:51.268479021 +0000 UTC m=+2283.731495181" watchObservedRunningTime="2026-05-25 13:43:51.271920726 +0000 UTC m=+2283.734936883"
May 25 13:43:59.196506 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:43:59.196455    2557 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-86bd7ccwkq"
May 25 13:43:59.196979 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:43:59.196561    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-86bd7ccwkq"
May 25 13:43:59.199324 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:43:59.199300    2557 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-86bd7ccwkq"
May 25 13:43:59.278431 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:43:59.278404    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-86bd7ccwkq"
May 25 13:44:21.286224 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:44:21.286190    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-86bd7ccwkq"
May 25 13:45:48.232133 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:45:48.232013    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vqnvc_d21bdf8b-d0ab-48f2-8736-54e92947dd53/ovn-acl-logging/0.log"
May 25 13:45:48.239807 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:45:48.239784    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vqnvc_d21bdf8b-d0ab-48f2-8736-54e92947dd53/ovn-acl-logging/0.log"
May 25 13:50:26.375949 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:50:26.375909    2557 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve/llmisvc-controller-manager-6644479599-4x2cl"]
May 25 13:50:26.376469 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:50:26.376313    2557 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve/llmisvc-controller-manager-6644479599-4x2cl" podUID="639b620f-34a7-42f4-866e-f769f4d753c4" containerName="manager" containerID="cri-o://f39a7593571cf57b9380e8c4fc7e1ca5fac968fe467ff05bd9bbf2b1c0592595" gracePeriod=30
May 25 13:50:26.626628 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:50:26.626550    2557 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-6644479599-4x2cl"
May 25 13:50:26.698000 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:50:26.697966    2557 generic.go:358] "Generic (PLEG): container finished" podID="639b620f-34a7-42f4-866e-f769f4d753c4" containerID="f39a7593571cf57b9380e8c4fc7e1ca5fac968fe467ff05bd9bbf2b1c0592595" exitCode=0
May 25 13:50:26.698179 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:50:26.698018    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-6644479599-4x2cl" event={"ID":"639b620f-34a7-42f4-866e-f769f4d753c4","Type":"ContainerDied","Data":"f39a7593571cf57b9380e8c4fc7e1ca5fac968fe467ff05bd9bbf2b1c0592595"}
May 25 13:50:26.698179 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:50:26.698031    2557 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-6644479599-4x2cl"
May 25 13:50:26.698179 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:50:26.698041    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-6644479599-4x2cl" event={"ID":"639b620f-34a7-42f4-866e-f769f4d753c4","Type":"ContainerDied","Data":"066e315d275d89fc50246da0e71055d255b4de0535d62c5ea98d34b0554149c3"}
May 25 13:50:26.698179 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:50:26.698057    2557 scope.go:117] "RemoveContainer" containerID="f39a7593571cf57b9380e8c4fc7e1ca5fac968fe467ff05bd9bbf2b1c0592595"
May 25 13:50:26.708768 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:50:26.708741    2557 scope.go:117] "RemoveContainer" containerID="f39a7593571cf57b9380e8c4fc7e1ca5fac968fe467ff05bd9bbf2b1c0592595"
May 25 13:50:26.709105 ip-10-0-143-235 kubenswrapper[2557]: E0525 13:50:26.709077    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f39a7593571cf57b9380e8c4fc7e1ca5fac968fe467ff05bd9bbf2b1c0592595\": container with ID starting with f39a7593571cf57b9380e8c4fc7e1ca5fac968fe467ff05bd9bbf2b1c0592595 not found: ID does not exist" containerID="f39a7593571cf57b9380e8c4fc7e1ca5fac968fe467ff05bd9bbf2b1c0592595"
May 25 13:50:26.709250 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:50:26.709114    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f39a7593571cf57b9380e8c4fc7e1ca5fac968fe467ff05bd9bbf2b1c0592595"} err="failed to get container status \"f39a7593571cf57b9380e8c4fc7e1ca5fac968fe467ff05bd9bbf2b1c0592595\": rpc error: code = NotFound desc = could not find container \"f39a7593571cf57b9380e8c4fc7e1ca5fac968fe467ff05bd9bbf2b1c0592595\": container with ID starting with f39a7593571cf57b9380e8c4fc7e1ca5fac968fe467ff05bd9bbf2b1c0592595 not found: ID does not exist"
May 25 13:50:26.790759 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:50:26.790726    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/639b620f-34a7-42f4-866e-f769f4d753c4-cert\") pod \"639b620f-34a7-42f4-866e-f769f4d753c4\" (UID: \"639b620f-34a7-42f4-866e-f769f4d753c4\") "
May 25 13:50:26.790950 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:50:26.790778    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-74tpc\" (UniqueName: \"kubernetes.io/projected/639b620f-34a7-42f4-866e-f769f4d753c4-kube-api-access-74tpc\") pod \"639b620f-34a7-42f4-866e-f769f4d753c4\" (UID: \"639b620f-34a7-42f4-866e-f769f4d753c4\") "
May 25 13:50:26.792991 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:50:26.792951    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/639b620f-34a7-42f4-866e-f769f4d753c4-kube-api-access-74tpc" (OuterVolumeSpecName: "kube-api-access-74tpc") pod "639b620f-34a7-42f4-866e-f769f4d753c4" (UID: "639b620f-34a7-42f4-866e-f769f4d753c4"). InnerVolumeSpecName "kube-api-access-74tpc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 13:50:26.793112 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:50:26.793003    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/639b620f-34a7-42f4-866e-f769f4d753c4-cert" (OuterVolumeSpecName: "cert") pod "639b620f-34a7-42f4-866e-f769f4d753c4" (UID: "639b620f-34a7-42f4-866e-f769f4d753c4"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 13:50:26.892411 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:50:26.892285    2557 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/639b620f-34a7-42f4-866e-f769f4d753c4-cert\") on node \"ip-10-0-143-235.ec2.internal\" DevicePath \"\""
May 25 13:50:26.892411 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:50:26.892316    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-74tpc\" (UniqueName: \"kubernetes.io/projected/639b620f-34a7-42f4-866e-f769f4d753c4-kube-api-access-74tpc\") on node \"ip-10-0-143-235.ec2.internal\" DevicePath \"\""
May 25 13:50:27.021731 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:50:27.021691    2557 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve/llmisvc-controller-manager-6644479599-4x2cl"]
May 25 13:50:27.026342 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:50:27.026299    2557 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve/llmisvc-controller-manager-6644479599-4x2cl"]
May 25 13:50:28.150568 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:50:28.150533    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="639b620f-34a7-42f4-866e-f769f4d753c4" path="/var/lib/kubelet/pods/639b620f-34a7-42f4-866e-f769f4d753c4/volumes"
May 25 13:50:48.261195 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:50:48.261089    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vqnvc_d21bdf8b-d0ab-48f2-8736-54e92947dd53/ovn-acl-logging/0.log"
May 25 13:50:48.267470 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:50:48.267448    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vqnvc_d21bdf8b-d0ab-48f2-8736-54e92947dd53/ovn-acl-logging/0.log"
May 25 13:55:48.289154 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:55:48.289028    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vqnvc_d21bdf8b-d0ab-48f2-8736-54e92947dd53/ovn-acl-logging/0.log"
May 25 13:55:48.294011 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:55:48.293990    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vqnvc_d21bdf8b-d0ab-48f2-8736-54e92947dd53/ovn-acl-logging/0.log"
May 25 13:58:46.640796 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:58:46.640759    2557 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-86bd7ccwkq"]
May 25 13:58:46.641478 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:58:46.641164    2557 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-86bd7ccwkq" podUID="b29dde20-ae27-43e1-80ae-de916e45fc1e" containerName="main" containerID="cri-o://e6e54ceb9cf7e143f61f3b2c25e0f12d9b3b0bbfd06fddbd84a99c3d89081b11" gracePeriod=30
May 25 13:58:46.641580 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:58:46.641246    2557 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-86bd7ccwkq" podUID="b29dde20-ae27-43e1-80ae-de916e45fc1e" containerName="tokenizer" containerID="cri-o://12dbe62cbc7580b292a41d3dffeda340f1526b70f9a39d62d37179297975ae10" gracePeriod=30
May 25 13:58:47.505289 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:58:47.505252    2557 generic.go:358] "Generic (PLEG): container finished" podID="b29dde20-ae27-43e1-80ae-de916e45fc1e" containerID="e6e54ceb9cf7e143f61f3b2c25e0f12d9b3b0bbfd06fddbd84a99c3d89081b11" exitCode=0
May 25 13:58:47.505479 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:58:47.505331    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-86bd7ccwkq" event={"ID":"b29dde20-ae27-43e1-80ae-de916e45fc1e","Type":"ContainerDied","Data":"e6e54ceb9cf7e143f61f3b2c25e0f12d9b3b0bbfd06fddbd84a99c3d89081b11"}
May 25 13:58:48.011365 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:58:48.011313    2557 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-86bd7ccwkq"
May 25 13:58:48.026224 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:58:48.026185    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7rccg\" (UniqueName: \"kubernetes.io/projected/b29dde20-ae27-43e1-80ae-de916e45fc1e-kube-api-access-7rccg\") pod \"b29dde20-ae27-43e1-80ae-de916e45fc1e\" (UID: \"b29dde20-ae27-43e1-80ae-de916e45fc1e\") "
May 25 13:58:48.026413 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:58:48.026265    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/b29dde20-ae27-43e1-80ae-de916e45fc1e-tokenizer-cache\") pod \"b29dde20-ae27-43e1-80ae-de916e45fc1e\" (UID: \"b29dde20-ae27-43e1-80ae-de916e45fc1e\") "
May 25 13:58:48.026413 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:58:48.026306    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/b29dde20-ae27-43e1-80ae-de916e45fc1e-tokenizer-tmp\") pod \"b29dde20-ae27-43e1-80ae-de916e45fc1e\" (UID: \"b29dde20-ae27-43e1-80ae-de916e45fc1e\") "
May 25 13:58:48.026413 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:58:48.026381    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/b29dde20-ae27-43e1-80ae-de916e45fc1e-tokenizer-uds\") pod \"b29dde20-ae27-43e1-80ae-de916e45fc1e\" (UID: \"b29dde20-ae27-43e1-80ae-de916e45fc1e\") "
May 25 13:58:48.026413 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:58:48.026414    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b29dde20-ae27-43e1-80ae-de916e45fc1e-tls-certs\") pod \"b29dde20-ae27-43e1-80ae-de916e45fc1e\" (UID: \"b29dde20-ae27-43e1-80ae-de916e45fc1e\") "
May 25 13:58:48.026637 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:58:48.026437    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b29dde20-ae27-43e1-80ae-de916e45fc1e-kserve-provision-location\") pod \"b29dde20-ae27-43e1-80ae-de916e45fc1e\" (UID: \"b29dde20-ae27-43e1-80ae-de916e45fc1e\") "
May 25 13:58:48.026637 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:58:48.026597    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b29dde20-ae27-43e1-80ae-de916e45fc1e-tokenizer-cache" (OuterVolumeSpecName: "tokenizer-cache") pod "b29dde20-ae27-43e1-80ae-de916e45fc1e" (UID: "b29dde20-ae27-43e1-80ae-de916e45fc1e"). InnerVolumeSpecName "tokenizer-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 13:58:48.026739 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:58:48.026661    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b29dde20-ae27-43e1-80ae-de916e45fc1e-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "b29dde20-ae27-43e1-80ae-de916e45fc1e" (UID: "b29dde20-ae27-43e1-80ae-de916e45fc1e"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 13:58:48.026795 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:58:48.026782    2557 reconciler_common.go:299] "Volume detached for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/b29dde20-ae27-43e1-80ae-de916e45fc1e-tokenizer-cache\") on node \"ip-10-0-143-235.ec2.internal\" DevicePath \"\""
May 25 13:58:48.026854 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:58:48.026801    2557 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/b29dde20-ae27-43e1-80ae-de916e45fc1e-tokenizer-uds\") on node \"ip-10-0-143-235.ec2.internal\" DevicePath \"\""
May 25 13:58:48.026910 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:58:48.026877    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b29dde20-ae27-43e1-80ae-de916e45fc1e-tokenizer-tmp" (OuterVolumeSpecName: "tokenizer-tmp") pod "b29dde20-ae27-43e1-80ae-de916e45fc1e" (UID: "b29dde20-ae27-43e1-80ae-de916e45fc1e"). InnerVolumeSpecName "tokenizer-tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 13:58:48.027313 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:58:48.027289    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b29dde20-ae27-43e1-80ae-de916e45fc1e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b29dde20-ae27-43e1-80ae-de916e45fc1e" (UID: "b29dde20-ae27-43e1-80ae-de916e45fc1e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 13:58:48.028587 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:58:48.028555    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b29dde20-ae27-43e1-80ae-de916e45fc1e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b29dde20-ae27-43e1-80ae-de916e45fc1e" (UID: "b29dde20-ae27-43e1-80ae-de916e45fc1e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 13:58:48.028889 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:58:48.028874    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b29dde20-ae27-43e1-80ae-de916e45fc1e-kube-api-access-7rccg" (OuterVolumeSpecName: "kube-api-access-7rccg") pod "b29dde20-ae27-43e1-80ae-de916e45fc1e" (UID: "b29dde20-ae27-43e1-80ae-de916e45fc1e"). InnerVolumeSpecName "kube-api-access-7rccg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 13:58:48.127269 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:58:48.127167    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7rccg\" (UniqueName: \"kubernetes.io/projected/b29dde20-ae27-43e1-80ae-de916e45fc1e-kube-api-access-7rccg\") on node \"ip-10-0-143-235.ec2.internal\" DevicePath \"\""
May 25 13:58:48.127269 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:58:48.127205    2557 reconciler_common.go:299] "Volume detached for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/b29dde20-ae27-43e1-80ae-de916e45fc1e-tokenizer-tmp\") on node \"ip-10-0-143-235.ec2.internal\" DevicePath \"\""
May 25 13:58:48.127269 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:58:48.127216    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b29dde20-ae27-43e1-80ae-de916e45fc1e-tls-certs\") on node \"ip-10-0-143-235.ec2.internal\" DevicePath \"\""
May 25 13:58:48.127269 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:58:48.127224    2557 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b29dde20-ae27-43e1-80ae-de916e45fc1e-kserve-provision-location\") on node \"ip-10-0-143-235.ec2.internal\" DevicePath \"\""
May 25 13:58:48.360334 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:58:48.360302    2557 scope.go:117] "RemoveContainer" containerID="12dbe62cbc7580b292a41d3dffeda340f1526b70f9a39d62d37179297975ae10"
May 25 13:58:48.369094 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:58:48.369068    2557 scope.go:117] "RemoveContainer" containerID="7eea61e441b0d8fe44aa1726b7039d413511406bd03173aa939b725f645010e8"
May 25 13:58:48.377868 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:58:48.377844    2557 scope.go:117] "RemoveContainer" containerID="e6e54ceb9cf7e143f61f3b2c25e0f12d9b3b0bbfd06fddbd84a99c3d89081b11"
May 25 13:58:48.508149 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:58:48.508112    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-86bd7ccwkq" event={"ID":"b29dde20-ae27-43e1-80ae-de916e45fc1e","Type":"ContainerDied","Data":"12dbe62cbc7580b292a41d3dffeda340f1526b70f9a39d62d37179297975ae10"}
May 25 13:58:48.508149 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:58:48.508135    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-86bd7ccwkq"
May 25 13:58:48.508149 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:58:48.508157    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-86bd7ccwkq" event={"ID":"b29dde20-ae27-43e1-80ae-de916e45fc1e","Type":"ContainerDied","Data":"0d0234ee5f526b060214b20baf8ee01a772b64c3814db112e282d812a474c94f"}
May 25 13:58:48.547475 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:58:48.547432    2557 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-86bd7ccwkq"]
May 25 13:58:48.553079 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:58:48.553052    2557 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-86bd7ccwkq"]
May 25 13:58:50.150185 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:58:50.150152    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b29dde20-ae27-43e1-80ae-de916e45fc1e" path="/var/lib/kubelet/pods/b29dde20-ae27-43e1-80ae-de916e45fc1e/volumes"
May 25 13:58:55.652172 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:58:55.652139    2557 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-75b4cd78fqqhdn"]
May 25 13:58:55.652751 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:58:55.652638    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b29dde20-ae27-43e1-80ae-de916e45fc1e" containerName="storage-initializer"
May 25 13:58:55.652751 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:58:55.652656    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="b29dde20-ae27-43e1-80ae-de916e45fc1e" containerName="storage-initializer"
May 25 13:58:55.652751 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:58:55.652669    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b29dde20-ae27-43e1-80ae-de916e45fc1e" containerName="tokenizer"
May 25 13:58:55.652751 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:58:55.652676    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="b29dde20-ae27-43e1-80ae-de916e45fc1e" containerName="tokenizer"
May 25 13:58:55.652751 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:58:55.652689    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b29dde20-ae27-43e1-80ae-de916e45fc1e" containerName="main"
May 25 13:58:55.652751 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:58:55.652694    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="b29dde20-ae27-43e1-80ae-de916e45fc1e" containerName="main"
May 25 13:58:55.652751 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:58:55.652705    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="639b620f-34a7-42f4-866e-f769f4d753c4" containerName="manager"
May 25 13:58:55.652751 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:58:55.652710    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="639b620f-34a7-42f4-866e-f769f4d753c4" containerName="manager"
May 25 13:58:55.653132 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:58:55.652776    2557 memory_manager.go:356] "RemoveStaleState removing state" podUID="639b620f-34a7-42f4-866e-f769f4d753c4" containerName="manager"
May 25 13:58:55.653132 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:58:55.652785    2557 memory_manager.go:356] "RemoveStaleState removing state" podUID="b29dde20-ae27-43e1-80ae-de916e45fc1e" containerName="main"
May 25 13:58:55.653132 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:58:55.652792    2557 memory_manager.go:356] "RemoveStaleState removing state" podUID="b29dde20-ae27-43e1-80ae-de916e45fc1e" containerName="tokenizer"
May 25 13:58:55.657694 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:58:55.657677    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-75b4cd78fqqhdn"
May 25 13:58:55.660413 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:58:55.660386    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
May 25 13:58:55.660572 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:58:55.660555    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-8blwv\""
May 25 13:58:55.661484 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:58:55.661470    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"router-with-refs-pd-test-epp-sa-dockercfg-nbkhv\""
May 25 13:58:55.661543 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:58:55.661531    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"router-with-refs-pd-test-kserve-self-signed-certs\""
May 25 13:58:55.661599 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:58:55.661578    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
May 25 13:58:55.665699 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:58:55.665678    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-75b4cd78fqqhdn"]
May 25 13:58:55.696035 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:58:55.695997    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2lxm4\" (UniqueName: \"kubernetes.io/projected/c28eac9d-4db4-4ecf-b7f0-39835ce68014-kube-api-access-2lxm4\") pod \"router-with-refs-pd-test-kserve-router-scheduler-75b4cd78fqqhdn\" (UID: \"c28eac9d-4db4-4ecf-b7f0-39835ce68014\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-75b4cd78fqqhdn"
May 25 13:58:55.696215 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:58:55.696051    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c28eac9d-4db4-4ecf-b7f0-39835ce68014-tls-certs\") pod \"router-with-refs-pd-test-kserve-router-scheduler-75b4cd78fqqhdn\" (UID: \"c28eac9d-4db4-4ecf-b7f0-39835ce68014\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-75b4cd78fqqhdn"
May 25 13:58:55.696215 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:58:55.696144    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/c28eac9d-4db4-4ecf-b7f0-39835ce68014-tokenizer-cache\") pod \"router-with-refs-pd-test-kserve-router-scheduler-75b4cd78fqqhdn\" (UID: \"c28eac9d-4db4-4ecf-b7f0-39835ce68014\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-75b4cd78fqqhdn"
May 25 13:58:55.696215 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:58:55.696179    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/c28eac9d-4db4-4ecf-b7f0-39835ce68014-tokenizer-tmp\") pod \"router-with-refs-pd-test-kserve-router-scheduler-75b4cd78fqqhdn\" (UID: \"c28eac9d-4db4-4ecf-b7f0-39835ce68014\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-75b4cd78fqqhdn"
May 25 13:58:55.696328 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:58:55.696230    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/c28eac9d-4db4-4ecf-b7f0-39835ce68014-tokenizer-uds\") pod \"router-with-refs-pd-test-kserve-router-scheduler-75b4cd78fqqhdn\" (UID: \"c28eac9d-4db4-4ecf-b7f0-39835ce68014\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-75b4cd78fqqhdn"
May 25 13:58:55.696328 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:58:55.696260    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c28eac9d-4db4-4ecf-b7f0-39835ce68014-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-router-scheduler-75b4cd78fqqhdn\" (UID: \"c28eac9d-4db4-4ecf-b7f0-39835ce68014\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-75b4cd78fqqhdn"
May 25 13:58:55.797179 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:58:55.797137    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c28eac9d-4db4-4ecf-b7f0-39835ce68014-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-router-scheduler-75b4cd78fqqhdn\" (UID: \"c28eac9d-4db4-4ecf-b7f0-39835ce68014\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-75b4cd78fqqhdn"
May 25 13:58:55.797379 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:58:55.797279    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-2lxm4\" (UniqueName: \"kubernetes.io/projected/c28eac9d-4db4-4ecf-b7f0-39835ce68014-kube-api-access-2lxm4\") pod \"router-with-refs-pd-test-kserve-router-scheduler-75b4cd78fqqhdn\" (UID: \"c28eac9d-4db4-4ecf-b7f0-39835ce68014\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-75b4cd78fqqhdn"
May 25 13:58:55.797379 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:58:55.797314    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c28eac9d-4db4-4ecf-b7f0-39835ce68014-tls-certs\") pod \"router-with-refs-pd-test-kserve-router-scheduler-75b4cd78fqqhdn\" (UID: \"c28eac9d-4db4-4ecf-b7f0-39835ce68014\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-75b4cd78fqqhdn"
May 25 13:58:55.797464 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:58:55.797382    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/c28eac9d-4db4-4ecf-b7f0-39835ce68014-tokenizer-cache\") pod \"router-with-refs-pd-test-kserve-router-scheduler-75b4cd78fqqhdn\" (UID: \"c28eac9d-4db4-4ecf-b7f0-39835ce68014\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-75b4cd78fqqhdn"
May 25 13:58:55.797464 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:58:55.797407    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/c28eac9d-4db4-4ecf-b7f0-39835ce68014-tokenizer-tmp\") pod \"router-with-refs-pd-test-kserve-router-scheduler-75b4cd78fqqhdn\" (UID: \"c28eac9d-4db4-4ecf-b7f0-39835ce68014\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-75b4cd78fqqhdn"
May 25 13:58:55.797464 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:58:55.797426    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/c28eac9d-4db4-4ecf-b7f0-39835ce68014-tokenizer-uds\") pod \"router-with-refs-pd-test-kserve-router-scheduler-75b4cd78fqqhdn\" (UID: \"c28eac9d-4db4-4ecf-b7f0-39835ce68014\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-75b4cd78fqqhdn"
May 25 13:58:55.797589 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:58:55.797534    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c28eac9d-4db4-4ecf-b7f0-39835ce68014-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-router-scheduler-75b4cd78fqqhdn\" (UID: \"c28eac9d-4db4-4ecf-b7f0-39835ce68014\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-75b4cd78fqqhdn"
May 25 13:58:55.797736 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:58:55.797719    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/c28eac9d-4db4-4ecf-b7f0-39835ce68014-tokenizer-cache\") pod \"router-with-refs-pd-test-kserve-router-scheduler-75b4cd78fqqhdn\" (UID: \"c28eac9d-4db4-4ecf-b7f0-39835ce68014\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-75b4cd78fqqhdn"
May 25 13:58:55.797782 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:58:55.797765    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/c28eac9d-4db4-4ecf-b7f0-39835ce68014-tokenizer-tmp\") pod \"router-with-refs-pd-test-kserve-router-scheduler-75b4cd78fqqhdn\" (UID: \"c28eac9d-4db4-4ecf-b7f0-39835ce68014\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-75b4cd78fqqhdn"
May 25 13:58:55.797828 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:58:55.797782    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/c28eac9d-4db4-4ecf-b7f0-39835ce68014-tokenizer-uds\") pod \"router-with-refs-pd-test-kserve-router-scheduler-75b4cd78fqqhdn\" (UID: \"c28eac9d-4db4-4ecf-b7f0-39835ce68014\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-75b4cd78fqqhdn"
May 25 13:58:55.799904 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:58:55.799885    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c28eac9d-4db4-4ecf-b7f0-39835ce68014-tls-certs\") pod \"router-with-refs-pd-test-kserve-router-scheduler-75b4cd78fqqhdn\" (UID: \"c28eac9d-4db4-4ecf-b7f0-39835ce68014\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-75b4cd78fqqhdn"
May 25 13:58:55.805988 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:58:55.805954    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2lxm4\" (UniqueName: \"kubernetes.io/projected/c28eac9d-4db4-4ecf-b7f0-39835ce68014-kube-api-access-2lxm4\") pod \"router-with-refs-pd-test-kserve-router-scheduler-75b4cd78fqqhdn\" (UID: \"c28eac9d-4db4-4ecf-b7f0-39835ce68014\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-75b4cd78fqqhdn"
May 25 13:58:55.968170 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:58:55.968064    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-75b4cd78fqqhdn"
May 25 13:58:56.104748 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:58:56.104699    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-75b4cd78fqqhdn"]
May 25 13:58:56.107943 ip-10-0-143-235 kubenswrapper[2557]: W0525 13:58:56.107912    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc28eac9d_4db4_4ecf_b7f0_39835ce68014.slice/crio-2cc67a5a0602c1f7b7d7bec21d95c1ee3757ac592d07d4a45a6a3e9732232af8 WatchSource:0}: Error finding container 2cc67a5a0602c1f7b7d7bec21d95c1ee3757ac592d07d4a45a6a3e9732232af8: Status 404 returned error can't find the container with id 2cc67a5a0602c1f7b7d7bec21d95c1ee3757ac592d07d4a45a6a3e9732232af8
May 25 13:58:56.109827 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:58:56.109800    2557 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 25 13:58:56.539086 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:58:56.539036    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-75b4cd78fqqhdn" event={"ID":"c28eac9d-4db4-4ecf-b7f0-39835ce68014","Type":"ContainerStarted","Data":"0d0339df450c710df48c740b9e4fc0774251ede682e9126411f1931bbb344691"}
May 25 13:58:56.539086 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:58:56.539080    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-75b4cd78fqqhdn" event={"ID":"c28eac9d-4db4-4ecf-b7f0-39835ce68014","Type":"ContainerStarted","Data":"2cc67a5a0602c1f7b7d7bec21d95c1ee3757ac592d07d4a45a6a3e9732232af8"}
May 25 13:58:57.543175 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:58:57.543134    2557 generic.go:358] "Generic (PLEG): container finished" podID="c28eac9d-4db4-4ecf-b7f0-39835ce68014" containerID="0d0339df450c710df48c740b9e4fc0774251ede682e9126411f1931bbb344691" exitCode=0
May 25 13:58:57.543586 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:58:57.543211    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-75b4cd78fqqhdn" event={"ID":"c28eac9d-4db4-4ecf-b7f0-39835ce68014","Type":"ContainerDied","Data":"0d0339df450c710df48c740b9e4fc0774251ede682e9126411f1931bbb344691"}
May 25 13:58:58.552703 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:58:58.552614    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-75b4cd78fqqhdn" event={"ID":"c28eac9d-4db4-4ecf-b7f0-39835ce68014","Type":"ContainerStarted","Data":"5ebe74b1ad6bbfe31205478b51929137bf0c5589660904f53bef8a6ce8698aaa"}
May 25 13:58:58.552703 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:58:58.552665    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-75b4cd78fqqhdn" event={"ID":"c28eac9d-4db4-4ecf-b7f0-39835ce68014","Type":"ContainerStarted","Data":"5e552ddc1c7eb9f7a6f34f94c1b54dd1d20c6eb612aeb577401644938677282f"}
May 25 13:58:58.553194 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:58:58.552832    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-75b4cd78fqqhdn"
May 25 13:58:58.575575 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:58:58.575526    2557 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-75b4cd78fqqhdn" podStartSLOduration=3.5755105289999998 podStartE2EDuration="3.575510529s" podCreationTimestamp="2026-05-25 13:58:55 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 13:58:58.574390175 +0000 UTC m=+3191.037406331" watchObservedRunningTime="2026-05-25 13:58:58.575510529 +0000 UTC m=+3191.038526686"
May 25 13:59:05.968519 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:59:05.968475    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-75b4cd78fqqhdn"
May 25 13:59:05.968519 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:59:05.968523    2557 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-75b4cd78fqqhdn"
May 25 13:59:05.971562 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:59:05.971533    2557 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-75b4cd78fqqhdn"
May 25 13:59:06.583400 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:59:06.583367    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-75b4cd78fqqhdn"
May 25 13:59:27.586843 ip-10-0-143-235 kubenswrapper[2557]: I0525 13:59:27.586811    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-75b4cd78fqqhdn"
May 25 14:00:48.314562 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:00:48.314533    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vqnvc_d21bdf8b-d0ab-48f2-8736-54e92947dd53/ovn-acl-logging/0.log"
May 25 14:00:48.319571 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:00:48.319545    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vqnvc_d21bdf8b-d0ab-48f2-8736-54e92947dd53/ovn-acl-logging/0.log"
May 25 14:01:36.300597 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:01:36.300559    2557 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-75b4cd78fqqhdn"]
May 25 14:01:36.301214 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:01:36.300897    2557 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-75b4cd78fqqhdn" podUID="c28eac9d-4db4-4ecf-b7f0-39835ce68014" containerName="main" containerID="cri-o://5e552ddc1c7eb9f7a6f34f94c1b54dd1d20c6eb612aeb577401644938677282f" gracePeriod=30
May 25 14:01:36.301214 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:01:36.300941    2557 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-75b4cd78fqqhdn" podUID="c28eac9d-4db4-4ecf-b7f0-39835ce68014" containerName="tokenizer" containerID="cri-o://5ebe74b1ad6bbfe31205478b51929137bf0c5589660904f53bef8a6ce8698aaa" gracePeriod=30
May 25 14:01:36.582862 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:01:36.582754    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-75b4cd78fqqhdn" podUID="c28eac9d-4db4-4ecf-b7f0-39835ce68014" containerName="tokenizer" probeResult="failure" output="Get \"http://10.134.0.39:8082/healthz\": dial tcp 10.134.0.39:8082: connect: connection refused"
May 25 14:01:37.149105 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:01:37.149068    2557 generic.go:358] "Generic (PLEG): container finished" podID="c28eac9d-4db4-4ecf-b7f0-39835ce68014" containerID="5e552ddc1c7eb9f7a6f34f94c1b54dd1d20c6eb612aeb577401644938677282f" exitCode=0
May 25 14:01:37.149286 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:01:37.149135    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-75b4cd78fqqhdn" event={"ID":"c28eac9d-4db4-4ecf-b7f0-39835ce68014","Type":"ContainerDied","Data":"5e552ddc1c7eb9f7a6f34f94c1b54dd1d20c6eb612aeb577401644938677282f"}
May 25 14:01:37.650943 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:01:37.650917    2557 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-75b4cd78fqqhdn"
May 25 14:01:37.749948 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:01:37.749854    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/c28eac9d-4db4-4ecf-b7f0-39835ce68014-tokenizer-tmp\") pod \"c28eac9d-4db4-4ecf-b7f0-39835ce68014\" (UID: \"c28eac9d-4db4-4ecf-b7f0-39835ce68014\") "
May 25 14:01:37.749948 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:01:37.749907    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/c28eac9d-4db4-4ecf-b7f0-39835ce68014-tokenizer-uds\") pod \"c28eac9d-4db4-4ecf-b7f0-39835ce68014\" (UID: \"c28eac9d-4db4-4ecf-b7f0-39835ce68014\") "
May 25 14:01:37.750174 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:01:37.750010    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c28eac9d-4db4-4ecf-b7f0-39835ce68014-kserve-provision-location\") pod \"c28eac9d-4db4-4ecf-b7f0-39835ce68014\" (UID: \"c28eac9d-4db4-4ecf-b7f0-39835ce68014\") "
May 25 14:01:37.750174 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:01:37.750041    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2lxm4\" (UniqueName: \"kubernetes.io/projected/c28eac9d-4db4-4ecf-b7f0-39835ce68014-kube-api-access-2lxm4\") pod \"c28eac9d-4db4-4ecf-b7f0-39835ce68014\" (UID: \"c28eac9d-4db4-4ecf-b7f0-39835ce68014\") "
May 25 14:01:37.750174 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:01:37.750075    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c28eac9d-4db4-4ecf-b7f0-39835ce68014-tls-certs\") pod \"c28eac9d-4db4-4ecf-b7f0-39835ce68014\" (UID: \"c28eac9d-4db4-4ecf-b7f0-39835ce68014\") "
May 25 14:01:37.750174 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:01:37.750132    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/c28eac9d-4db4-4ecf-b7f0-39835ce68014-tokenizer-cache\") pod \"c28eac9d-4db4-4ecf-b7f0-39835ce68014\" (UID: \"c28eac9d-4db4-4ecf-b7f0-39835ce68014\") "
May 25 14:01:37.750402 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:01:37.750181    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c28eac9d-4db4-4ecf-b7f0-39835ce68014-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "c28eac9d-4db4-4ecf-b7f0-39835ce68014" (UID: "c28eac9d-4db4-4ecf-b7f0-39835ce68014"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 14:01:37.750402 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:01:37.750254    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c28eac9d-4db4-4ecf-b7f0-39835ce68014-tokenizer-tmp" (OuterVolumeSpecName: "tokenizer-tmp") pod "c28eac9d-4db4-4ecf-b7f0-39835ce68014" (UID: "c28eac9d-4db4-4ecf-b7f0-39835ce68014"). InnerVolumeSpecName "tokenizer-tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 14:01:37.750510 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:01:37.750462    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c28eac9d-4db4-4ecf-b7f0-39835ce68014-tokenizer-cache" (OuterVolumeSpecName: "tokenizer-cache") pod "c28eac9d-4db4-4ecf-b7f0-39835ce68014" (UID: "c28eac9d-4db4-4ecf-b7f0-39835ce68014"). InnerVolumeSpecName "tokenizer-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 14:01:37.750587 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:01:37.750569    2557 reconciler_common.go:299] "Volume detached for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/c28eac9d-4db4-4ecf-b7f0-39835ce68014-tokenizer-cache\") on node \"ip-10-0-143-235.ec2.internal\" DevicePath \"\""
May 25 14:01:37.750618 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:01:37.750594    2557 reconciler_common.go:299] "Volume detached for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/c28eac9d-4db4-4ecf-b7f0-39835ce68014-tokenizer-tmp\") on node \"ip-10-0-143-235.ec2.internal\" DevicePath \"\""
May 25 14:01:37.750618 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:01:37.750610    2557 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/c28eac9d-4db4-4ecf-b7f0-39835ce68014-tokenizer-uds\") on node \"ip-10-0-143-235.ec2.internal\" DevicePath \"\""
May 25 14:01:37.750858 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:01:37.750834    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c28eac9d-4db4-4ecf-b7f0-39835ce68014-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c28eac9d-4db4-4ecf-b7f0-39835ce68014" (UID: "c28eac9d-4db4-4ecf-b7f0-39835ce68014"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 14:01:37.752216 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:01:37.752186    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c28eac9d-4db4-4ecf-b7f0-39835ce68014-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c28eac9d-4db4-4ecf-b7f0-39835ce68014" (UID: "c28eac9d-4db4-4ecf-b7f0-39835ce68014"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 14:01:37.752332 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:01:37.752306    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c28eac9d-4db4-4ecf-b7f0-39835ce68014-kube-api-access-2lxm4" (OuterVolumeSpecName: "kube-api-access-2lxm4") pod "c28eac9d-4db4-4ecf-b7f0-39835ce68014" (UID: "c28eac9d-4db4-4ecf-b7f0-39835ce68014"). InnerVolumeSpecName "kube-api-access-2lxm4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 14:01:37.852093 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:01:37.852037    2557 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c28eac9d-4db4-4ecf-b7f0-39835ce68014-kserve-provision-location\") on node \"ip-10-0-143-235.ec2.internal\" DevicePath \"\""
May 25 14:01:37.852093 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:01:37.852089    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2lxm4\" (UniqueName: \"kubernetes.io/projected/c28eac9d-4db4-4ecf-b7f0-39835ce68014-kube-api-access-2lxm4\") on node \"ip-10-0-143-235.ec2.internal\" DevicePath \"\""
May 25 14:01:37.852093 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:01:37.852101    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c28eac9d-4db4-4ecf-b7f0-39835ce68014-tls-certs\") on node \"ip-10-0-143-235.ec2.internal\" DevicePath \"\""
May 25 14:01:38.153428 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:01:38.153389    2557 generic.go:358] "Generic (PLEG): container finished" podID="c28eac9d-4db4-4ecf-b7f0-39835ce68014" containerID="5ebe74b1ad6bbfe31205478b51929137bf0c5589660904f53bef8a6ce8698aaa" exitCode=0
May 25 14:01:38.153645 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:01:38.153463    2557 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-75b4cd78fqqhdn"
May 25 14:01:38.153645 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:01:38.153463    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-75b4cd78fqqhdn" event={"ID":"c28eac9d-4db4-4ecf-b7f0-39835ce68014","Type":"ContainerDied","Data":"5ebe74b1ad6bbfe31205478b51929137bf0c5589660904f53bef8a6ce8698aaa"}
May 25 14:01:38.153645 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:01:38.153578    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-75b4cd78fqqhdn" event={"ID":"c28eac9d-4db4-4ecf-b7f0-39835ce68014","Type":"ContainerDied","Data":"2cc67a5a0602c1f7b7d7bec21d95c1ee3757ac592d07d4a45a6a3e9732232af8"}
May 25 14:01:38.153645 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:01:38.153596    2557 scope.go:117] "RemoveContainer" containerID="5ebe74b1ad6bbfe31205478b51929137bf0c5589660904f53bef8a6ce8698aaa"
May 25 14:01:38.163657 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:01:38.163636    2557 scope.go:117] "RemoveContainer" containerID="5e552ddc1c7eb9f7a6f34f94c1b54dd1d20c6eb612aeb577401644938677282f"
May 25 14:01:38.171397 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:01:38.171375    2557 scope.go:117] "RemoveContainer" containerID="0d0339df450c710df48c740b9e4fc0774251ede682e9126411f1931bbb344691"
May 25 14:01:38.176565 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:01:38.176537    2557 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-75b4cd78fqqhdn"]
May 25 14:01:38.179789 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:01:38.179763    2557 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-75b4cd78fqqhdn"]
May 25 14:01:38.180246 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:01:38.180231    2557 scope.go:117] "RemoveContainer" containerID="5ebe74b1ad6bbfe31205478b51929137bf0c5589660904f53bef8a6ce8698aaa"
May 25 14:01:38.180558 ip-10-0-143-235 kubenswrapper[2557]: E0525 14:01:38.180537    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ebe74b1ad6bbfe31205478b51929137bf0c5589660904f53bef8a6ce8698aaa\": container with ID starting with 5ebe74b1ad6bbfe31205478b51929137bf0c5589660904f53bef8a6ce8698aaa not found: ID does not exist" containerID="5ebe74b1ad6bbfe31205478b51929137bf0c5589660904f53bef8a6ce8698aaa"
May 25 14:01:38.180635 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:01:38.180572    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ebe74b1ad6bbfe31205478b51929137bf0c5589660904f53bef8a6ce8698aaa"} err="failed to get container status \"5ebe74b1ad6bbfe31205478b51929137bf0c5589660904f53bef8a6ce8698aaa\": rpc error: code = NotFound desc = could not find container \"5ebe74b1ad6bbfe31205478b51929137bf0c5589660904f53bef8a6ce8698aaa\": container with ID starting with 5ebe74b1ad6bbfe31205478b51929137bf0c5589660904f53bef8a6ce8698aaa not found: ID does not exist"
May 25 14:01:38.180635 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:01:38.180602    2557 scope.go:117] "RemoveContainer" containerID="5e552ddc1c7eb9f7a6f34f94c1b54dd1d20c6eb612aeb577401644938677282f"
May 25 14:01:38.180869 ip-10-0-143-235 kubenswrapper[2557]: E0525 14:01:38.180851    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e552ddc1c7eb9f7a6f34f94c1b54dd1d20c6eb612aeb577401644938677282f\": container with ID starting with 5e552ddc1c7eb9f7a6f34f94c1b54dd1d20c6eb612aeb577401644938677282f not found: ID does not exist" containerID="5e552ddc1c7eb9f7a6f34f94c1b54dd1d20c6eb612aeb577401644938677282f"
May 25 14:01:38.180911 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:01:38.180882    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e552ddc1c7eb9f7a6f34f94c1b54dd1d20c6eb612aeb577401644938677282f"} err="failed to get container status \"5e552ddc1c7eb9f7a6f34f94c1b54dd1d20c6eb612aeb577401644938677282f\": rpc error: code = NotFound desc = could not find container \"5e552ddc1c7eb9f7a6f34f94c1b54dd1d20c6eb612aeb577401644938677282f\": container with ID starting with 5e552ddc1c7eb9f7a6f34f94c1b54dd1d20c6eb612aeb577401644938677282f not found: ID does not exist"
May 25 14:01:38.180911 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:01:38.180899    2557 scope.go:117] "RemoveContainer" containerID="0d0339df450c710df48c740b9e4fc0774251ede682e9126411f1931bbb344691"
May 25 14:01:38.181129 ip-10-0-143-235 kubenswrapper[2557]: E0525 14:01:38.181113    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d0339df450c710df48c740b9e4fc0774251ede682e9126411f1931bbb344691\": container with ID starting with 0d0339df450c710df48c740b9e4fc0774251ede682e9126411f1931bbb344691 not found: ID does not exist" containerID="0d0339df450c710df48c740b9e4fc0774251ede682e9126411f1931bbb344691"
May 25 14:01:38.181177 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:01:38.181136    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d0339df450c710df48c740b9e4fc0774251ede682e9126411f1931bbb344691"} err="failed to get container status \"0d0339df450c710df48c740b9e4fc0774251ede682e9126411f1931bbb344691\": rpc error: code = NotFound desc = could not find container \"0d0339df450c710df48c740b9e4fc0774251ede682e9126411f1931bbb344691\": container with ID starting with 0d0339df450c710df48c740b9e4fc0774251ede682e9126411f1931bbb344691 not found: ID does not exist"
May 25 14:01:38.586555 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:01:38.586508    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-75b4cd78fqqhdn" podUID="c28eac9d-4db4-4ecf-b7f0-39835ce68014" containerName="main" probeResult="failure" output="timeout: failed to connect service \"10.134.0.39:9003\" within 1s: context deadline exceeded"
May 25 14:01:38.586952 ip-10-0-143-235 kubenswrapper[2557]: W0525 14:01:38.586589    2557 logging.go:55] [core] [Channel #1304 SubChannel #1305]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.39:9003", ServerName: "10.134.0.39:9003", }. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.39:9003: operation was canceled"
May 25 14:01:40.151082 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:01:40.151036    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c28eac9d-4db4-4ecf-b7f0-39835ce68014" path="/var/lib/kubelet/pods/c28eac9d-4db4-4ecf-b7f0-39835ce68014/volumes"
May 25 14:05:48.339452 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:05:48.339326    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vqnvc_d21bdf8b-d0ab-48f2-8736-54e92947dd53/ovn-acl-logging/0.log"
May 25 14:05:48.345573 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:05:48.345130    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vqnvc_d21bdf8b-d0ab-48f2-8736-54e92947dd53/ovn-acl-logging/0.log"
May 25 14:06:11.353723 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:06:11.353659    2557 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1"]
May 25 14:06:11.354105 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:06:11.354089    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c28eac9d-4db4-4ecf-b7f0-39835ce68014" containerName="main"
May 25 14:06:11.354156 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:06:11.354106    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="c28eac9d-4db4-4ecf-b7f0-39835ce68014" containerName="main"
May 25 14:06:11.354156 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:06:11.354115    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c28eac9d-4db4-4ecf-b7f0-39835ce68014" containerName="tokenizer"
May 25 14:06:11.354156 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:06:11.354121    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="c28eac9d-4db4-4ecf-b7f0-39835ce68014" containerName="tokenizer"
May 25 14:06:11.354156 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:06:11.354130    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c28eac9d-4db4-4ecf-b7f0-39835ce68014" containerName="storage-initializer"
May 25 14:06:11.354156 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:06:11.354136    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="c28eac9d-4db4-4ecf-b7f0-39835ce68014" containerName="storage-initializer"
May 25 14:06:11.354317 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:06:11.354200    2557 memory_manager.go:356] "RemoveStaleState removing state" podUID="c28eac9d-4db4-4ecf-b7f0-39835ce68014" containerName="tokenizer"
May 25 14:06:11.354317 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:06:11.354209    2557 memory_manager.go:356] "RemoveStaleState removing state" podUID="c28eac9d-4db4-4ecf-b7f0-39835ce68014" containerName="main"
May 25 14:06:11.357130 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:06:11.357113    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1"
May 25 14:06:11.362465 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:06:11.362423    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
May 25 14:06:11.364115 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:06:11.363868    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-fb-opt-125m-route-f312f5-cb7fb8cf-dockercfg-x2vg8\""
May 25 14:06:11.364115 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:06:11.363888    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
May 25 14:06:11.364115 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:06:11.363880    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv4e643bc258191ffc517a31cd1d0ddd27-kserve-self-signed-certs\""
May 25 14:06:11.364281 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:06:11.364149    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-8blwv\""
May 25 14:06:11.371745 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:06:11.371719    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1"]
May 25 14:06:11.405502 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:06:11.405455    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/271337d0-9579-4768-aff1-1e27a1e6245a-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1\" (UID: \"271337d0-9579-4768-aff1-1e27a1e6245a\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1"
May 25 14:06:11.405502 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:06:11.405507    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/271337d0-9579-4768-aff1-1e27a1e6245a-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1\" (UID: \"271337d0-9579-4768-aff1-1e27a1e6245a\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1"
May 25 14:06:11.405761 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:06:11.405577    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/271337d0-9579-4768-aff1-1e27a1e6245a-home\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1\" (UID: \"271337d0-9579-4768-aff1-1e27a1e6245a\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1"
May 25 14:06:11.405761 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:06:11.405654    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/271337d0-9579-4768-aff1-1e27a1e6245a-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1\" (UID: \"271337d0-9579-4768-aff1-1e27a1e6245a\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1"
May 25 14:06:11.405761 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:06:11.405705    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/271337d0-9579-4768-aff1-1e27a1e6245a-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1\" (UID: \"271337d0-9579-4768-aff1-1e27a1e6245a\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1"
May 25 14:06:11.405761 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:06:11.405736    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkgv2\" (UniqueName: \"kubernetes.io/projected/271337d0-9579-4768-aff1-1e27a1e6245a-kube-api-access-pkgv2\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1\" (UID: \"271337d0-9579-4768-aff1-1e27a1e6245a\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1"
May 25 14:06:11.405915 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:06:11.405793    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/271337d0-9579-4768-aff1-1e27a1e6245a-dshm\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1\" (UID: \"271337d0-9579-4768-aff1-1e27a1e6245a\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1"
May 25 14:06:11.451640 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:06:11.451604    2557 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schehl5k2"]
May 25 14:06:11.455178 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:06:11.455153    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schehl5k2"
May 25 14:06:11.461772 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:06:11.461561    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-fb-opt-125m-route-f312f5ec-epp-sa-dockercfg-nshtb\""
May 25 14:06:11.470756 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:06:11.470718    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schehl5k2"]
May 25 14:06:11.506492 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:06:11.506459    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/271337d0-9579-4768-aff1-1e27a1e6245a-dshm\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1\" (UID: \"271337d0-9579-4768-aff1-1e27a1e6245a\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1"
May 25 14:06:11.506660 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:06:11.506502    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/271337d0-9579-4768-aff1-1e27a1e6245a-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1\" (UID: \"271337d0-9579-4768-aff1-1e27a1e6245a\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1"
May 25 14:06:11.506660 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:06:11.506613    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/271337d0-9579-4768-aff1-1e27a1e6245a-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1\" (UID: \"271337d0-9579-4768-aff1-1e27a1e6245a\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1"
May 25 14:06:11.506772 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:06:11.506666    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/271337d0-9579-4768-aff1-1e27a1e6245a-home\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1\" (UID: \"271337d0-9579-4768-aff1-1e27a1e6245a\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1"
May 25 14:06:11.506772 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:06:11.506708    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/49d9476f-cc07-43a5-b2b4-0b1b048a8923-kserve-provision-location\") pod \"llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schehl5k2\" (UID: \"49d9476f-cc07-43a5-b2b4-0b1b048a8923\") " pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schehl5k2"
May 25 14:06:11.506772 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:06:11.506737    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/271337d0-9579-4768-aff1-1e27a1e6245a-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1\" (UID: \"271337d0-9579-4768-aff1-1e27a1e6245a\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1"
May 25 14:06:11.506772 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:06:11.506766    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/49d9476f-cc07-43a5-b2b4-0b1b048a8923-tokenizer-tmp\") pod \"llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schehl5k2\" (UID: \"49d9476f-cc07-43a5-b2b4-0b1b048a8923\") " pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schehl5k2"
May 25 14:06:11.506987 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:06:11.506773    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/271337d0-9579-4768-aff1-1e27a1e6245a-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1\" (UID: \"271337d0-9579-4768-aff1-1e27a1e6245a\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1"
May 25 14:06:11.506987 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:06:11.506790    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/49d9476f-cc07-43a5-b2b4-0b1b048a8923-tokenizer-cache\") pod \"llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schehl5k2\" (UID: \"49d9476f-cc07-43a5-b2b4-0b1b048a8923\") " pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schehl5k2"
May 25 14:06:11.506987 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:06:11.506829    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/271337d0-9579-4768-aff1-1e27a1e6245a-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1\" (UID: \"271337d0-9579-4768-aff1-1e27a1e6245a\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1"
May 25 14:06:11.506987 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:06:11.506902    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-pkgv2\" (UniqueName: \"kubernetes.io/projected/271337d0-9579-4768-aff1-1e27a1e6245a-kube-api-access-pkgv2\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1\" (UID: \"271337d0-9579-4768-aff1-1e27a1e6245a\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1"
May 25 14:06:11.506987 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:06:11.506928    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/271337d0-9579-4768-aff1-1e27a1e6245a-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1\" (UID: \"271337d0-9579-4768-aff1-1e27a1e6245a\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1"
May 25 14:06:11.507307 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:06:11.506934    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/49d9476f-cc07-43a5-b2b4-0b1b048a8923-tls-certs\") pod \"llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schehl5k2\" (UID: \"49d9476f-cc07-43a5-b2b4-0b1b048a8923\") " pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schehl5k2"
May 25 14:06:11.507307 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:06:11.507027    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/271337d0-9579-4768-aff1-1e27a1e6245a-home\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1\" (UID: \"271337d0-9579-4768-aff1-1e27a1e6245a\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1"
May 25 14:06:11.507307 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:06:11.507062    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/49d9476f-cc07-43a5-b2b4-0b1b048a8923-tokenizer-uds\") pod \"llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schehl5k2\" (UID: \"49d9476f-cc07-43a5-b2b4-0b1b048a8923\") " pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schehl5k2"
May 25 14:06:11.507307 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:06:11.507072    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/271337d0-9579-4768-aff1-1e27a1e6245a-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1\" (UID: \"271337d0-9579-4768-aff1-1e27a1e6245a\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1"
May 25 14:06:11.507307 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:06:11.507103    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmn67\" (UniqueName: \"kubernetes.io/projected/49d9476f-cc07-43a5-b2b4-0b1b048a8923-kube-api-access-lmn67\") pod \"llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schehl5k2\" (UID: \"49d9476f-cc07-43a5-b2b4-0b1b048a8923\") " pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schehl5k2"
May 25 14:06:11.508801 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:06:11.508778    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/271337d0-9579-4768-aff1-1e27a1e6245a-dshm\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1\" (UID: \"271337d0-9579-4768-aff1-1e27a1e6245a\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1"
May 25 14:06:11.508917 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:06:11.508899    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/271337d0-9579-4768-aff1-1e27a1e6245a-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1\" (UID: \"271337d0-9579-4768-aff1-1e27a1e6245a\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1"
May 25 14:06:11.522898 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:06:11.522870    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkgv2\" (UniqueName: \"kubernetes.io/projected/271337d0-9579-4768-aff1-1e27a1e6245a-kube-api-access-pkgv2\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1\" (UID: \"271337d0-9579-4768-aff1-1e27a1e6245a\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1"
May 25 14:06:11.608435 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:06:11.608315    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/49d9476f-cc07-43a5-b2b4-0b1b048a8923-kserve-provision-location\") pod \"llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schehl5k2\" (UID: \"49d9476f-cc07-43a5-b2b4-0b1b048a8923\") " pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schehl5k2"
May 25 14:06:11.608435 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:06:11.608404    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/49d9476f-cc07-43a5-b2b4-0b1b048a8923-tokenizer-tmp\") pod \"llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schehl5k2\" (UID: \"49d9476f-cc07-43a5-b2b4-0b1b048a8923\") " pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schehl5k2"
May 25 14:06:11.608435 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:06:11.608426    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/49d9476f-cc07-43a5-b2b4-0b1b048a8923-tokenizer-cache\") pod \"llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schehl5k2\" (UID: \"49d9476f-cc07-43a5-b2b4-0b1b048a8923\") " pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schehl5k2"
May 25 14:06:11.608738 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:06:11.608463    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/49d9476f-cc07-43a5-b2b4-0b1b048a8923-tls-certs\") pod \"llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schehl5k2\" (UID: \"49d9476f-cc07-43a5-b2b4-0b1b048a8923\") " pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schehl5k2"
May 25 14:06:11.608738 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:06:11.608488    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/49d9476f-cc07-43a5-b2b4-0b1b048a8923-tokenizer-uds\") pod \"llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schehl5k2\" (UID: \"49d9476f-cc07-43a5-b2b4-0b1b048a8923\") " pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schehl5k2"
May 25 14:06:11.608738 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:06:11.608507    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-lmn67\" (UniqueName: \"kubernetes.io/projected/49d9476f-cc07-43a5-b2b4-0b1b048a8923-kube-api-access-lmn67\") pod \"llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schehl5k2\" (UID: \"49d9476f-cc07-43a5-b2b4-0b1b048a8923\") " pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schehl5k2"
May 25 14:06:11.608915 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:06:11.608787    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/49d9476f-cc07-43a5-b2b4-0b1b048a8923-kserve-provision-location\") pod \"llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schehl5k2\" (UID: \"49d9476f-cc07-43a5-b2b4-0b1b048a8923\") " pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schehl5k2"
May 25 14:06:11.608915 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:06:11.608815    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/49d9476f-cc07-43a5-b2b4-0b1b048a8923-tokenizer-tmp\") pod \"llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schehl5k2\" (UID: \"49d9476f-cc07-43a5-b2b4-0b1b048a8923\") " pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schehl5k2"
May 25 14:06:11.608915 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:06:11.608848    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/49d9476f-cc07-43a5-b2b4-0b1b048a8923-tokenizer-cache\") pod \"llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schehl5k2\" (UID: \"49d9476f-cc07-43a5-b2b4-0b1b048a8923\") " pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schehl5k2"
May 25 14:06:11.609029 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:06:11.608940    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/49d9476f-cc07-43a5-b2b4-0b1b048a8923-tokenizer-uds\") pod \"llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schehl5k2\" (UID: \"49d9476f-cc07-43a5-b2b4-0b1b048a8923\") " pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schehl5k2"
May 25 14:06:11.610961 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:06:11.610934    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/49d9476f-cc07-43a5-b2b4-0b1b048a8923-tls-certs\") pod \"llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schehl5k2\" (UID: \"49d9476f-cc07-43a5-b2b4-0b1b048a8923\") " pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schehl5k2"
May 25 14:06:11.619233 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:06:11.619204    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmn67\" (UniqueName: \"kubernetes.io/projected/49d9476f-cc07-43a5-b2b4-0b1b048a8923-kube-api-access-lmn67\") pod \"llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schehl5k2\" (UID: \"49d9476f-cc07-43a5-b2b4-0b1b048a8923\") " pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schehl5k2"
May 25 14:06:11.668174 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:06:11.668138    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1"
May 25 14:06:11.764627 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:06:11.764583    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schehl5k2"
May 25 14:06:11.814598 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:06:11.814543    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1"]
May 25 14:06:11.817722 ip-10-0-143-235 kubenswrapper[2557]: W0525 14:06:11.817685    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod271337d0_9579_4768_aff1_1e27a1e6245a.slice/crio-a3d255cb7b11308bfdcbd399a5004d557e6da7d893cd96f3ee40d7ac77e0bad1 WatchSource:0}: Error finding container a3d255cb7b11308bfdcbd399a5004d557e6da7d893cd96f3ee40d7ac77e0bad1: Status 404 returned error can't find the container with id a3d255cb7b11308bfdcbd399a5004d557e6da7d893cd96f3ee40d7ac77e0bad1
May 25 14:06:11.819752 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:06:11.819732    2557 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 25 14:06:11.923604 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:06:11.923574    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schehl5k2"]
May 25 14:06:11.925604 ip-10-0-143-235 kubenswrapper[2557]: W0525 14:06:11.925577    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49d9476f_cc07_43a5_b2b4_0b1b048a8923.slice/crio-90d98d905ff7eeb2f17f45c7d2666c52993a76ba46b4635ba2af3592f5320937 WatchSource:0}: Error finding container 90d98d905ff7eeb2f17f45c7d2666c52993a76ba46b4635ba2af3592f5320937: Status 404 returned error can't find the container with id 90d98d905ff7eeb2f17f45c7d2666c52993a76ba46b4635ba2af3592f5320937
May 25 14:06:12.122408 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:06:12.122302    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schehl5k2" event={"ID":"49d9476f-cc07-43a5-b2b4-0b1b048a8923","Type":"ContainerStarted","Data":"52331db2281b19e1d8d02320456087b571016a8ab7c5aee5ca617540e4e3e163"}
May 25 14:06:12.122408 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:06:12.122359    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schehl5k2" event={"ID":"49d9476f-cc07-43a5-b2b4-0b1b048a8923","Type":"ContainerStarted","Data":"90d98d905ff7eeb2f17f45c7d2666c52993a76ba46b4635ba2af3592f5320937"}
May 25 14:06:12.123725 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:06:12.123695    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1" event={"ID":"271337d0-9579-4768-aff1-1e27a1e6245a","Type":"ContainerStarted","Data":"374e878e381d3e684397240f2b1e8d05ac9d90e2529f5c4768ca2aa518474b78"}
May 25 14:06:12.123725 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:06:12.123724    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1" event={"ID":"271337d0-9579-4768-aff1-1e27a1e6245a","Type":"ContainerStarted","Data":"a3d255cb7b11308bfdcbd399a5004d557e6da7d893cd96f3ee40d7ac77e0bad1"}
May 25 14:06:13.129678 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:06:13.129636    2557 generic.go:358] "Generic (PLEG): container finished" podID="49d9476f-cc07-43a5-b2b4-0b1b048a8923" containerID="52331db2281b19e1d8d02320456087b571016a8ab7c5aee5ca617540e4e3e163" exitCode=0
May 25 14:06:13.130071 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:06:13.129757    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schehl5k2" event={"ID":"49d9476f-cc07-43a5-b2b4-0b1b048a8923","Type":"ContainerDied","Data":"52331db2281b19e1d8d02320456087b571016a8ab7c5aee5ca617540e4e3e163"}
May 25 14:06:14.138225 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:06:14.138142    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schehl5k2" event={"ID":"49d9476f-cc07-43a5-b2b4-0b1b048a8923","Type":"ContainerStarted","Data":"813c1629a284161aa0df2d1fe3ba99372a612a38ad86b9182d3043762b3aeb87"}
May 25 14:06:14.138667 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:06:14.138249    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schehl5k2"
May 25 14:06:14.138667 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:06:14.138267    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schehl5k2" event={"ID":"49d9476f-cc07-43a5-b2b4-0b1b048a8923","Type":"ContainerStarted","Data":"3f9a636f3c7e02103972f087289b9dac7e9a15bbf1acda47be3a1fc0fcbf7430"}
May 25 14:06:14.163928 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:06:14.163863    2557 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schehl5k2" podStartSLOduration=3.163840327 podStartE2EDuration="3.163840327s" podCreationTimestamp="2026-05-25 14:06:11 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 14:06:14.162061192 +0000 UTC m=+3626.625077345" watchObservedRunningTime="2026-05-25 14:06:14.163840327 +0000 UTC m=+3626.626856485"
May 25 14:06:16.155456 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:06:16.155420    2557 generic.go:358] "Generic (PLEG): container finished" podID="271337d0-9579-4768-aff1-1e27a1e6245a" containerID="374e878e381d3e684397240f2b1e8d05ac9d90e2529f5c4768ca2aa518474b78" exitCode=0
May 25 14:06:16.155889 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:06:16.155470    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1" event={"ID":"271337d0-9579-4768-aff1-1e27a1e6245a","Type":"ContainerDied","Data":"374e878e381d3e684397240f2b1e8d05ac9d90e2529f5c4768ca2aa518474b78"}
May 25 14:06:21.765019 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:06:21.764980    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schehl5k2"
May 25 14:06:21.765673 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:06:21.765648    2557 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schehl5k2"
May 25 14:06:21.767054 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:06:21.766854    2557 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schehl5k2" podUID="49d9476f-cc07-43a5-b2b4-0b1b048a8923" containerName="tokenizer" probeResult="failure" output="Get \"http://10.134.0.41:8082/healthz\": dial tcp 10.134.0.41:8082: connect: connection refused"
May 25 14:06:31.766984 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:06:31.766950    2557 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schehl5k2"
May 25 14:06:31.768437 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:06:31.768408    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schehl5k2"
May 25 14:06:52.235236 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:06:52.235203    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schehl5k2"
May 25 14:06:56.329063 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:06:56.329024    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1" event={"ID":"271337d0-9579-4768-aff1-1e27a1e6245a","Type":"ContainerStarted","Data":"10c9f4ea915276dc39934f4e1d8666997523c5a5ad7f860af0cbc92eb4aa61a8"}
May 25 14:06:56.367225 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:06:56.367167    2557 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1" podStartSLOduration=5.371844363 podStartE2EDuration="45.367150248s" podCreationTimestamp="2026-05-25 14:06:11 +0000 UTC" firstStartedPulling="2026-05-25 14:06:16.156749534 +0000 UTC m=+3628.619765668" lastFinishedPulling="2026-05-25 14:06:56.152055405 +0000 UTC m=+3668.615071553" observedRunningTime="2026-05-25 14:06:56.364974793 +0000 UTC m=+3668.827990950" watchObservedRunningTime="2026-05-25 14:06:56.367150248 +0000 UTC m=+3668.830166404"
May 25 14:09:39.254543 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:09:39.254496    2557 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schehl5k2"]
May 25 14:09:39.255145 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:09:39.254908    2557 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schehl5k2" podUID="49d9476f-cc07-43a5-b2b4-0b1b048a8923" containerName="main" containerID="cri-o://3f9a636f3c7e02103972f087289b9dac7e9a15bbf1acda47be3a1fc0fcbf7430" gracePeriod=30
May 25 14:09:39.255145 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:09:39.255018    2557 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schehl5k2" podUID="49d9476f-cc07-43a5-b2b4-0b1b048a8923" containerName="tokenizer" containerID="cri-o://813c1629a284161aa0df2d1fe3ba99372a612a38ad86b9182d3043762b3aeb87" gracePeriod=30
May 25 14:09:39.955700 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:09:39.955662    2557 generic.go:358] "Generic (PLEG): container finished" podID="49d9476f-cc07-43a5-b2b4-0b1b048a8923" containerID="3f9a636f3c7e02103972f087289b9dac7e9a15bbf1acda47be3a1fc0fcbf7430" exitCode=0
May 25 14:09:39.955887 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:09:39.955738    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schehl5k2" event={"ID":"49d9476f-cc07-43a5-b2b4-0b1b048a8923","Type":"ContainerDied","Data":"3f9a636f3c7e02103972f087289b9dac7e9a15bbf1acda47be3a1fc0fcbf7430"}
May 25 14:09:40.612481 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:09:40.612442    2557 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schehl5k2"
May 25 14:09:40.713804 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:09:40.713710    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/49d9476f-cc07-43a5-b2b4-0b1b048a8923-tokenizer-uds\") pod \"49d9476f-cc07-43a5-b2b4-0b1b048a8923\" (UID: \"49d9476f-cc07-43a5-b2b4-0b1b048a8923\") "
May 25 14:09:40.713804 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:09:40.713793    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/49d9476f-cc07-43a5-b2b4-0b1b048a8923-kserve-provision-location\") pod \"49d9476f-cc07-43a5-b2b4-0b1b048a8923\" (UID: \"49d9476f-cc07-43a5-b2b4-0b1b048a8923\") "
May 25 14:09:40.714037 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:09:40.713826    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/49d9476f-cc07-43a5-b2b4-0b1b048a8923-tls-certs\") pod \"49d9476f-cc07-43a5-b2b4-0b1b048a8923\" (UID: \"49d9476f-cc07-43a5-b2b4-0b1b048a8923\") "
May 25 14:09:40.714037 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:09:40.713879    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/49d9476f-cc07-43a5-b2b4-0b1b048a8923-tokenizer-tmp\") pod \"49d9476f-cc07-43a5-b2b4-0b1b048a8923\" (UID: \"49d9476f-cc07-43a5-b2b4-0b1b048a8923\") "
May 25 14:09:40.714037 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:09:40.713905    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lmn67\" (UniqueName: \"kubernetes.io/projected/49d9476f-cc07-43a5-b2b4-0b1b048a8923-kube-api-access-lmn67\") pod \"49d9476f-cc07-43a5-b2b4-0b1b048a8923\" (UID: \"49d9476f-cc07-43a5-b2b4-0b1b048a8923\") "
May 25 14:09:40.714037 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:09:40.713954    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/49d9476f-cc07-43a5-b2b4-0b1b048a8923-tokenizer-cache\") pod \"49d9476f-cc07-43a5-b2b4-0b1b048a8923\" (UID: \"49d9476f-cc07-43a5-b2b4-0b1b048a8923\") "
May 25 14:09:40.714037 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:09:40.714030    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49d9476f-cc07-43a5-b2b4-0b1b048a8923-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "49d9476f-cc07-43a5-b2b4-0b1b048a8923" (UID: "49d9476f-cc07-43a5-b2b4-0b1b048a8923"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 14:09:40.714291 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:09:40.714232    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49d9476f-cc07-43a5-b2b4-0b1b048a8923-tokenizer-cache" (OuterVolumeSpecName: "tokenizer-cache") pod "49d9476f-cc07-43a5-b2b4-0b1b048a8923" (UID: "49d9476f-cc07-43a5-b2b4-0b1b048a8923"). InnerVolumeSpecName "tokenizer-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 14:09:40.714291 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:09:40.714248    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49d9476f-cc07-43a5-b2b4-0b1b048a8923-tokenizer-tmp" (OuterVolumeSpecName: "tokenizer-tmp") pod "49d9476f-cc07-43a5-b2b4-0b1b048a8923" (UID: "49d9476f-cc07-43a5-b2b4-0b1b048a8923"). InnerVolumeSpecName "tokenizer-tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 14:09:40.714291 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:09:40.714276    2557 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/49d9476f-cc07-43a5-b2b4-0b1b048a8923-tokenizer-uds\") on node \"ip-10-0-143-235.ec2.internal\" DevicePath \"\""
May 25 14:09:40.714680 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:09:40.714657    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49d9476f-cc07-43a5-b2b4-0b1b048a8923-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "49d9476f-cc07-43a5-b2b4-0b1b048a8923" (UID: "49d9476f-cc07-43a5-b2b4-0b1b048a8923"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 14:09:40.716117 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:09:40.716094    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/49d9476f-cc07-43a5-b2b4-0b1b048a8923-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "49d9476f-cc07-43a5-b2b4-0b1b048a8923" (UID: "49d9476f-cc07-43a5-b2b4-0b1b048a8923"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 14:09:40.716185 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:09:40.716160    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49d9476f-cc07-43a5-b2b4-0b1b048a8923-kube-api-access-lmn67" (OuterVolumeSpecName: "kube-api-access-lmn67") pod "49d9476f-cc07-43a5-b2b4-0b1b048a8923" (UID: "49d9476f-cc07-43a5-b2b4-0b1b048a8923"). InnerVolumeSpecName "kube-api-access-lmn67". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 14:09:40.815297 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:09:40.815253    2557 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/49d9476f-cc07-43a5-b2b4-0b1b048a8923-kserve-provision-location\") on node \"ip-10-0-143-235.ec2.internal\" DevicePath \"\""
May 25 14:09:40.815297 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:09:40.815291    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/49d9476f-cc07-43a5-b2b4-0b1b048a8923-tls-certs\") on node \"ip-10-0-143-235.ec2.internal\" DevicePath \"\""
May 25 14:09:40.815297 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:09:40.815304    2557 reconciler_common.go:299] "Volume detached for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/49d9476f-cc07-43a5-b2b4-0b1b048a8923-tokenizer-tmp\") on node \"ip-10-0-143-235.ec2.internal\" DevicePath \"\""
May 25 14:09:40.815570 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:09:40.815314    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lmn67\" (UniqueName: \"kubernetes.io/projected/49d9476f-cc07-43a5-b2b4-0b1b048a8923-kube-api-access-lmn67\") on node \"ip-10-0-143-235.ec2.internal\" DevicePath \"\""
May 25 14:09:40.815570 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:09:40.815323    2557 reconciler_common.go:299] "Volume detached for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/49d9476f-cc07-43a5-b2b4-0b1b048a8923-tokenizer-cache\") on node \"ip-10-0-143-235.ec2.internal\" DevicePath \"\""
May 25 14:09:40.952132 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:09:40.952093    2557 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1"]
May 25 14:09:40.952515 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:09:40.952390    2557 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1" podUID="271337d0-9579-4768-aff1-1e27a1e6245a" containerName="main" containerID="cri-o://10c9f4ea915276dc39934f4e1d8666997523c5a5ad7f860af0cbc92eb4aa61a8" gracePeriod=30
May 25 14:09:40.960873 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:09:40.960841    2557 generic.go:358] "Generic (PLEG): container finished" podID="49d9476f-cc07-43a5-b2b4-0b1b048a8923" containerID="813c1629a284161aa0df2d1fe3ba99372a612a38ad86b9182d3043762b3aeb87" exitCode=0
May 25 14:09:40.961071 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:09:40.960919    2557 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schehl5k2"
May 25 14:09:40.961071 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:09:40.960929    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schehl5k2" event={"ID":"49d9476f-cc07-43a5-b2b4-0b1b048a8923","Type":"ContainerDied","Data":"813c1629a284161aa0df2d1fe3ba99372a612a38ad86b9182d3043762b3aeb87"}
May 25 14:09:40.961071 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:09:40.960976    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schehl5k2" event={"ID":"49d9476f-cc07-43a5-b2b4-0b1b048a8923","Type":"ContainerDied","Data":"90d98d905ff7eeb2f17f45c7d2666c52993a76ba46b4635ba2af3592f5320937"}
May 25 14:09:40.961071 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:09:40.960998    2557 scope.go:117] "RemoveContainer" containerID="813c1629a284161aa0df2d1fe3ba99372a612a38ad86b9182d3043762b3aeb87"
May 25 14:09:40.970274 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:09:40.970248    2557 scope.go:117] "RemoveContainer" containerID="3f9a636f3c7e02103972f087289b9dac7e9a15bbf1acda47be3a1fc0fcbf7430"
May 25 14:09:40.979592 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:09:40.979569    2557 scope.go:117] "RemoveContainer" containerID="52331db2281b19e1d8d02320456087b571016a8ab7c5aee5ca617540e4e3e163"
May 25 14:09:40.984333 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:09:40.984301    2557 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schehl5k2"]
May 25 14:09:40.989476 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:09:40.989446    2557 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schehl5k2"]
May 25 14:09:40.989956 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:09:40.989939    2557 scope.go:117] "RemoveContainer" containerID="813c1629a284161aa0df2d1fe3ba99372a612a38ad86b9182d3043762b3aeb87"
May 25 14:09:40.990236 ip-10-0-143-235 kubenswrapper[2557]: E0525 14:09:40.990214    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"813c1629a284161aa0df2d1fe3ba99372a612a38ad86b9182d3043762b3aeb87\": container with ID starting with 813c1629a284161aa0df2d1fe3ba99372a612a38ad86b9182d3043762b3aeb87 not found: ID does not exist" containerID="813c1629a284161aa0df2d1fe3ba99372a612a38ad86b9182d3043762b3aeb87"
May 25 14:09:40.990305 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:09:40.990245    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"813c1629a284161aa0df2d1fe3ba99372a612a38ad86b9182d3043762b3aeb87"} err="failed to get container status \"813c1629a284161aa0df2d1fe3ba99372a612a38ad86b9182d3043762b3aeb87\": rpc error: code = NotFound desc = could not find container \"813c1629a284161aa0df2d1fe3ba99372a612a38ad86b9182d3043762b3aeb87\": container with ID starting with 813c1629a284161aa0df2d1fe3ba99372a612a38ad86b9182d3043762b3aeb87 not found: ID does not exist"
May 25 14:09:40.990305 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:09:40.990264    2557 scope.go:117] "RemoveContainer" containerID="3f9a636f3c7e02103972f087289b9dac7e9a15bbf1acda47be3a1fc0fcbf7430"
May 25 14:09:40.990526 ip-10-0-143-235 kubenswrapper[2557]: E0525 14:09:40.990510    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f9a636f3c7e02103972f087289b9dac7e9a15bbf1acda47be3a1fc0fcbf7430\": container with ID starting with 3f9a636f3c7e02103972f087289b9dac7e9a15bbf1acda47be3a1fc0fcbf7430 not found: ID does not exist" containerID="3f9a636f3c7e02103972f087289b9dac7e9a15bbf1acda47be3a1fc0fcbf7430"
May 25 14:09:40.990574 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:09:40.990532    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f9a636f3c7e02103972f087289b9dac7e9a15bbf1acda47be3a1fc0fcbf7430"} err="failed to get container status \"3f9a636f3c7e02103972f087289b9dac7e9a15bbf1acda47be3a1fc0fcbf7430\": rpc error: code = NotFound desc = could not find container \"3f9a636f3c7e02103972f087289b9dac7e9a15bbf1acda47be3a1fc0fcbf7430\": container with ID starting with 3f9a636f3c7e02103972f087289b9dac7e9a15bbf1acda47be3a1fc0fcbf7430 not found: ID does not exist"
May 25 14:09:40.990574 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:09:40.990547    2557 scope.go:117] "RemoveContainer" containerID="52331db2281b19e1d8d02320456087b571016a8ab7c5aee5ca617540e4e3e163"
May 25 14:09:40.990780 ip-10-0-143-235 kubenswrapper[2557]: E0525 14:09:40.990765    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"52331db2281b19e1d8d02320456087b571016a8ab7c5aee5ca617540e4e3e163\": container with ID starting with 52331db2281b19e1d8d02320456087b571016a8ab7c5aee5ca617540e4e3e163 not found: ID does not exist" containerID="52331db2281b19e1d8d02320456087b571016a8ab7c5aee5ca617540e4e3e163"
May 25 14:09:40.990822 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:09:40.990783    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52331db2281b19e1d8d02320456087b571016a8ab7c5aee5ca617540e4e3e163"} err="failed to get container status \"52331db2281b19e1d8d02320456087b571016a8ab7c5aee5ca617540e4e3e163\": rpc error: code = NotFound desc = could not find container \"52331db2281b19e1d8d02320456087b571016a8ab7c5aee5ca617540e4e3e163\": container with ID starting with 52331db2281b19e1d8d02320456087b571016a8ab7c5aee5ca617540e4e3e163 not found: ID does not exist"
May 25 14:09:41.705331 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:09:41.705304    2557 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1"
May 25 14:09:41.824739 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:09:41.824705    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/271337d0-9579-4768-aff1-1e27a1e6245a-home\") pod \"271337d0-9579-4768-aff1-1e27a1e6245a\" (UID: \"271337d0-9579-4768-aff1-1e27a1e6245a\") "
May 25 14:09:41.824928 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:09:41.824794    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/271337d0-9579-4768-aff1-1e27a1e6245a-model-cache\") pod \"271337d0-9579-4768-aff1-1e27a1e6245a\" (UID: \"271337d0-9579-4768-aff1-1e27a1e6245a\") "
May 25 14:09:41.824928 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:09:41.824822    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/271337d0-9579-4768-aff1-1e27a1e6245a-dshm\") pod \"271337d0-9579-4768-aff1-1e27a1e6245a\" (UID: \"271337d0-9579-4768-aff1-1e27a1e6245a\") "
May 25 14:09:41.824928 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:09:41.824846    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pkgv2\" (UniqueName: \"kubernetes.io/projected/271337d0-9579-4768-aff1-1e27a1e6245a-kube-api-access-pkgv2\") pod \"271337d0-9579-4768-aff1-1e27a1e6245a\" (UID: \"271337d0-9579-4768-aff1-1e27a1e6245a\") "
May 25 14:09:41.824928 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:09:41.824862    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/271337d0-9579-4768-aff1-1e27a1e6245a-kserve-provision-location\") pod \"271337d0-9579-4768-aff1-1e27a1e6245a\" (UID: \"271337d0-9579-4768-aff1-1e27a1e6245a\") "
May 25 14:09:41.824928 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:09:41.824881    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/271337d0-9579-4768-aff1-1e27a1e6245a-tmp-dir\") pod \"271337d0-9579-4768-aff1-1e27a1e6245a\" (UID: \"271337d0-9579-4768-aff1-1e27a1e6245a\") "
May 25 14:09:41.824928 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:09:41.824915    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/271337d0-9579-4768-aff1-1e27a1e6245a-tls-certs\") pod \"271337d0-9579-4768-aff1-1e27a1e6245a\" (UID: \"271337d0-9579-4768-aff1-1e27a1e6245a\") "
May 25 14:09:41.825291 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:09:41.825199    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/271337d0-9579-4768-aff1-1e27a1e6245a-model-cache" (OuterVolumeSpecName: "model-cache") pod "271337d0-9579-4768-aff1-1e27a1e6245a" (UID: "271337d0-9579-4768-aff1-1e27a1e6245a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 14:09:41.825543 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:09:41.825509    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/271337d0-9579-4768-aff1-1e27a1e6245a-home" (OuterVolumeSpecName: "home") pod "271337d0-9579-4768-aff1-1e27a1e6245a" (UID: "271337d0-9579-4768-aff1-1e27a1e6245a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 14:09:41.827017 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:09:41.826992    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/271337d0-9579-4768-aff1-1e27a1e6245a-dshm" (OuterVolumeSpecName: "dshm") pod "271337d0-9579-4768-aff1-1e27a1e6245a" (UID: "271337d0-9579-4768-aff1-1e27a1e6245a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 14:09:41.827121 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:09:41.827103    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/271337d0-9579-4768-aff1-1e27a1e6245a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "271337d0-9579-4768-aff1-1e27a1e6245a" (UID: "271337d0-9579-4768-aff1-1e27a1e6245a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 14:09:41.827287 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:09:41.827257    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/271337d0-9579-4768-aff1-1e27a1e6245a-kube-api-access-pkgv2" (OuterVolumeSpecName: "kube-api-access-pkgv2") pod "271337d0-9579-4768-aff1-1e27a1e6245a" (UID: "271337d0-9579-4768-aff1-1e27a1e6245a"). InnerVolumeSpecName "kube-api-access-pkgv2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 14:09:41.843793 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:09:41.843748    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/271337d0-9579-4768-aff1-1e27a1e6245a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "271337d0-9579-4768-aff1-1e27a1e6245a" (UID: "271337d0-9579-4768-aff1-1e27a1e6245a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 14:09:41.890321 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:09:41.890275    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/271337d0-9579-4768-aff1-1e27a1e6245a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "271337d0-9579-4768-aff1-1e27a1e6245a" (UID: "271337d0-9579-4768-aff1-1e27a1e6245a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 14:09:41.925830 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:09:41.925793    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/271337d0-9579-4768-aff1-1e27a1e6245a-model-cache\") on node \"ip-10-0-143-235.ec2.internal\" DevicePath \"\""
May 25 14:09:41.925830 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:09:41.925825    2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/271337d0-9579-4768-aff1-1e27a1e6245a-dshm\") on node \"ip-10-0-143-235.ec2.internal\" DevicePath \"\""
May 25 14:09:41.925830 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:09:41.925836    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pkgv2\" (UniqueName: \"kubernetes.io/projected/271337d0-9579-4768-aff1-1e27a1e6245a-kube-api-access-pkgv2\") on node \"ip-10-0-143-235.ec2.internal\" DevicePath \"\""
May 25 14:09:41.926047 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:09:41.925845    2557 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/271337d0-9579-4768-aff1-1e27a1e6245a-kserve-provision-location\") on node \"ip-10-0-143-235.ec2.internal\" DevicePath \"\""
May 25 14:09:41.926047 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:09:41.925856    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/271337d0-9579-4768-aff1-1e27a1e6245a-tmp-dir\") on node \"ip-10-0-143-235.ec2.internal\" DevicePath \"\""
May 25 14:09:41.926047 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:09:41.925871    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/271337d0-9579-4768-aff1-1e27a1e6245a-tls-certs\") on node \"ip-10-0-143-235.ec2.internal\" DevicePath \"\""
May 25 14:09:41.926047 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:09:41.925880    2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/271337d0-9579-4768-aff1-1e27a1e6245a-home\") on node \"ip-10-0-143-235.ec2.internal\" DevicePath \"\""
May 25 14:09:41.966876 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:09:41.966846    2557 generic.go:358] "Generic (PLEG): container finished" podID="271337d0-9579-4768-aff1-1e27a1e6245a" containerID="10c9f4ea915276dc39934f4e1d8666997523c5a5ad7f860af0cbc92eb4aa61a8" exitCode=0
May 25 14:09:41.967055 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:09:41.966917    2557 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1"
May 25 14:09:41.967055 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:09:41.966935    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1" event={"ID":"271337d0-9579-4768-aff1-1e27a1e6245a","Type":"ContainerDied","Data":"10c9f4ea915276dc39934f4e1d8666997523c5a5ad7f860af0cbc92eb4aa61a8"}
May 25 14:09:41.967055 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:09:41.966973    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1" event={"ID":"271337d0-9579-4768-aff1-1e27a1e6245a","Type":"ContainerDied","Data":"a3d255cb7b11308bfdcbd399a5004d557e6da7d893cd96f3ee40d7ac77e0bad1"}
May 25 14:09:41.967055 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:09:41.966986    2557 scope.go:117] "RemoveContainer" containerID="10c9f4ea915276dc39934f4e1d8666997523c5a5ad7f860af0cbc92eb4aa61a8"
May 25 14:09:41.976271 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:09:41.976250    2557 scope.go:117] "RemoveContainer" containerID="374e878e381d3e684397240f2b1e8d05ac9d90e2529f5c4768ca2aa518474b78"
May 25 14:09:41.989201 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:09:41.989168    2557 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1"]
May 25 14:09:41.992720 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:09:41.992689    2557 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1"]
May 25 14:09:41.993235 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:09:41.993219    2557 scope.go:117] "RemoveContainer" containerID="10c9f4ea915276dc39934f4e1d8666997523c5a5ad7f860af0cbc92eb4aa61a8"
May 25 14:09:41.993573 ip-10-0-143-235 kubenswrapper[2557]: E0525 14:09:41.993551    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10c9f4ea915276dc39934f4e1d8666997523c5a5ad7f860af0cbc92eb4aa61a8\": container with ID starting with 10c9f4ea915276dc39934f4e1d8666997523c5a5ad7f860af0cbc92eb4aa61a8 not found: ID does not exist" containerID="10c9f4ea915276dc39934f4e1d8666997523c5a5ad7f860af0cbc92eb4aa61a8"
May 25 14:09:41.993651 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:09:41.993579    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10c9f4ea915276dc39934f4e1d8666997523c5a5ad7f860af0cbc92eb4aa61a8"} err="failed to get container status \"10c9f4ea915276dc39934f4e1d8666997523c5a5ad7f860af0cbc92eb4aa61a8\": rpc error: code = NotFound desc = could not find container \"10c9f4ea915276dc39934f4e1d8666997523c5a5ad7f860af0cbc92eb4aa61a8\": container with ID starting with 10c9f4ea915276dc39934f4e1d8666997523c5a5ad7f860af0cbc92eb4aa61a8 not found: ID does not exist"
May 25 14:09:41.993651 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:09:41.993597    2557 scope.go:117] "RemoveContainer" containerID="374e878e381d3e684397240f2b1e8d05ac9d90e2529f5c4768ca2aa518474b78"
May 25 14:09:41.993841 ip-10-0-143-235 kubenswrapper[2557]: E0525 14:09:41.993823    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"374e878e381d3e684397240f2b1e8d05ac9d90e2529f5c4768ca2aa518474b78\": container with ID starting with 374e878e381d3e684397240f2b1e8d05ac9d90e2529f5c4768ca2aa518474b78 not found: ID does not exist" containerID="374e878e381d3e684397240f2b1e8d05ac9d90e2529f5c4768ca2aa518474b78"
May 25 14:09:41.993881 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:09:41.993845    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"374e878e381d3e684397240f2b1e8d05ac9d90e2529f5c4768ca2aa518474b78"} err="failed to get container status \"374e878e381d3e684397240f2b1e8d05ac9d90e2529f5c4768ca2aa518474b78\": rpc error: code = NotFound desc = could not find container \"374e878e381d3e684397240f2b1e8d05ac9d90e2529f5c4768ca2aa518474b78\": container with ID starting with 374e878e381d3e684397240f2b1e8d05ac9d90e2529f5c4768ca2aa518474b78 not found: ID does not exist"
May 25 14:09:42.150622 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:09:42.150543    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="271337d0-9579-4768-aff1-1e27a1e6245a" path="/var/lib/kubelet/pods/271337d0-9579-4768-aff1-1e27a1e6245a/volumes"
May 25 14:09:42.151024 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:09:42.151009    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="49d9476f-cc07-43a5-b2b4-0b1b048a8923" path="/var/lib/kubelet/pods/49d9476f-cc07-43a5-b2b4-0b1b048a8923/volumes"
May 25 14:10:10.196636 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:10:10.196542    2557 ???:1] "http: TLS handshake error from 10.0.135.71:59628: EOF"
May 25 14:10:10.203849 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:10:10.203816    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-68bd676465-8zsn9_3c78c0b9-30ff-420b-8fed-1e5537f4356d/authorino/0.log"
May 25 14:10:10.395826 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:10:10.395792    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-c7fb4c8d5-5nd5w_2cc8c2ee-258a-4fe8-ae11-a37e414c1266/manager/0.log"
May 25 14:10:12.723778 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:10:12.723743    2557 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-must-gather-tl2jt/must-gather-454lr"]
May 25 14:10:12.724162 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:10:12.724135    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="49d9476f-cc07-43a5-b2b4-0b1b048a8923" containerName="tokenizer"
May 25 14:10:12.724162 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:10:12.724148    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="49d9476f-cc07-43a5-b2b4-0b1b048a8923" containerName="tokenizer"
May 25 14:10:12.724162 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:10:12.724158    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="49d9476f-cc07-43a5-b2b4-0b1b048a8923" containerName="storage-initializer"
May 25 14:10:12.724162 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:10:12.724164    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="49d9476f-cc07-43a5-b2b4-0b1b048a8923" containerName="storage-initializer"
May 25 14:10:12.724295 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:10:12.724175    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="271337d0-9579-4768-aff1-1e27a1e6245a" containerName="storage-initializer"
May 25 14:10:12.724295 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:10:12.724182    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="271337d0-9579-4768-aff1-1e27a1e6245a" containerName="storage-initializer"
May 25 14:10:12.724295 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:10:12.724190    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="49d9476f-cc07-43a5-b2b4-0b1b048a8923" containerName="main"
May 25 14:10:12.724295 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:10:12.724195    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="49d9476f-cc07-43a5-b2b4-0b1b048a8923" containerName="main"
May 25 14:10:12.724295 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:10:12.724211    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="271337d0-9579-4768-aff1-1e27a1e6245a" containerName="main"
May 25 14:10:12.724295 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:10:12.724216    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="271337d0-9579-4768-aff1-1e27a1e6245a" containerName="main"
May 25 14:10:12.724295 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:10:12.724272    2557 memory_manager.go:356] "RemoveStaleState removing state" podUID="49d9476f-cc07-43a5-b2b4-0b1b048a8923" containerName="main"
May 25 14:10:12.724295 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:10:12.724280    2557 memory_manager.go:356] "RemoveStaleState removing state" podUID="49d9476f-cc07-43a5-b2b4-0b1b048a8923" containerName="tokenizer"
May 25 14:10:12.724295 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:10:12.724288    2557 memory_manager.go:356] "RemoveStaleState removing state" podUID="271337d0-9579-4768-aff1-1e27a1e6245a" containerName="main"
May 25 14:10:12.727210 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:10:12.727182    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-tl2jt/must-gather-454lr"
May 25 14:10:12.730641 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:10:12.730613    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-tl2jt\"/\"openshift-service-ca.crt\""
May 25 14:10:12.730848 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:10:12.730822    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-tl2jt\"/\"default-dockercfg-kltt4\""
May 25 14:10:12.731998 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:10:12.731980    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-tl2jt\"/\"kube-root-ca.crt\""
May 25 14:10:12.738224 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:10:12.738200    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-tl2jt/must-gather-454lr"]
May 25 14:10:12.817018 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:10:12.816972    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/0de808eb-6574-49eb-b369-39abbf3deb4c-must-gather-output\") pod \"must-gather-454lr\" (UID: \"0de808eb-6574-49eb-b369-39abbf3deb4c\") " pod="openshift-must-gather-tl2jt/must-gather-454lr"
May 25 14:10:12.817229 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:10:12.817034    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f49jb\" (UniqueName: \"kubernetes.io/projected/0de808eb-6574-49eb-b369-39abbf3deb4c-kube-api-access-f49jb\") pod \"must-gather-454lr\" (UID: \"0de808eb-6574-49eb-b369-39abbf3deb4c\") " pod="openshift-must-gather-tl2jt/must-gather-454lr"
May 25 14:10:12.917985 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:10:12.917943    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-f49jb\" (UniqueName: \"kubernetes.io/projected/0de808eb-6574-49eb-b369-39abbf3deb4c-kube-api-access-f49jb\") pod \"must-gather-454lr\" (UID: \"0de808eb-6574-49eb-b369-39abbf3deb4c\") " pod="openshift-must-gather-tl2jt/must-gather-454lr"
May 25 14:10:12.918165 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:10:12.918063    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/0de808eb-6574-49eb-b369-39abbf3deb4c-must-gather-output\") pod \"must-gather-454lr\" (UID: \"0de808eb-6574-49eb-b369-39abbf3deb4c\") " pod="openshift-must-gather-tl2jt/must-gather-454lr"
May 25 14:10:12.918488 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:10:12.918470    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/0de808eb-6574-49eb-b369-39abbf3deb4c-must-gather-output\") pod \"must-gather-454lr\" (UID: \"0de808eb-6574-49eb-b369-39abbf3deb4c\") " pod="openshift-must-gather-tl2jt/must-gather-454lr"
May 25 14:10:12.926947 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:10:12.926921    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f49jb\" (UniqueName: \"kubernetes.io/projected/0de808eb-6574-49eb-b369-39abbf3deb4c-kube-api-access-f49jb\") pod \"must-gather-454lr\" (UID: \"0de808eb-6574-49eb-b369-39abbf3deb4c\") " pod="openshift-must-gather-tl2jt/must-gather-454lr"
May 25 14:10:13.036743 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:10:13.036650    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-tl2jt/must-gather-454lr"
May 25 14:10:13.174492 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:10:13.174470    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-tl2jt/must-gather-454lr"]
May 25 14:10:13.176750 ip-10-0-143-235 kubenswrapper[2557]: W0525 14:10:13.176720    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0de808eb_6574_49eb_b369_39abbf3deb4c.slice/crio-a67c5f9808cc1411144ac9087d328ab05379d832d0d1ab132e8b02411868123a WatchSource:0}: Error finding container a67c5f9808cc1411144ac9087d328ab05379d832d0d1ab132e8b02411868123a: Status 404 returned error can't find the container with id a67c5f9808cc1411144ac9087d328ab05379d832d0d1ab132e8b02411868123a
May 25 14:10:14.088817 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:10:14.088725    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-tl2jt/must-gather-454lr" event={"ID":"0de808eb-6574-49eb-b369-39abbf3deb4c","Type":"ContainerStarted","Data":"8df4ce3105766bde0430528724830f721d6d2511687709d3317b4dd2956a8784"}
May 25 14:10:14.088817 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:10:14.088773    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-tl2jt/must-gather-454lr" event={"ID":"0de808eb-6574-49eb-b369-39abbf3deb4c","Type":"ContainerStarted","Data":"7a794cd2a198662e3c01840449707cf69547be53f3c46c7b053c7cbc6cb82007"}
May 25 14:10:14.088817 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:10:14.088788    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-tl2jt/must-gather-454lr" event={"ID":"0de808eb-6574-49eb-b369-39abbf3deb4c","Type":"ContainerStarted","Data":"a67c5f9808cc1411144ac9087d328ab05379d832d0d1ab132e8b02411868123a"}
May 25 14:10:14.106774 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:10:14.106720    2557 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-must-gather-tl2jt/must-gather-454lr" podStartSLOduration=1.45245973 podStartE2EDuration="2.106701593s" podCreationTimestamp="2026-05-25 14:10:12 +0000 UTC" firstStartedPulling="2026-05-25 14:10:13.178438756 +0000 UTC m=+3865.641454890" lastFinishedPulling="2026-05-25 14:10:13.83268061 +0000 UTC m=+3866.295696753" observedRunningTime="2026-05-25 14:10:14.104864508 +0000 UTC m=+3866.567880691" watchObservedRunningTime="2026-05-25 14:10:14.106701593 +0000 UTC m=+3866.569717753"
May 25 14:10:15.347503 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:10:15.347470    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-88vxt_81b07008-64d7-4948-a23d-4eaae2c55da5/global-pull-secret-syncer/0.log"
May 25 14:10:15.508798 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:10:15.508757    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-vwnkg_55e3ea80-f133-45b9-b3b6-0d1abdbb1ad0/konnectivity-agent/0.log"
May 25 14:10:15.619170 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:10:15.619090    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-143-235.ec2.internal_be380709ced6c0e0087779b84acc1278/haproxy/0.log"
May 25 14:10:19.504815 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:10:19.504775    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-68bd676465-8zsn9_3c78c0b9-30ff-420b-8fed-1e5537f4356d/authorino/0.log"
May 25 14:10:19.721905 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:10:19.721870    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-c7fb4c8d5-5nd5w_2cc8c2ee-258a-4fe8-ae11-a37e414c1266/manager/0.log"
May 25 14:10:20.776635 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:10:20.776601    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_5f764e8d-9e15-478e-8e6b-50ebf8290b0c/alertmanager/0.log"
May 25 14:10:20.803126 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:10:20.803092    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_5f764e8d-9e15-478e-8e6b-50ebf8290b0c/config-reloader/0.log"
May 25 14:10:20.828970 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:10:20.828933    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_5f764e8d-9e15-478e-8e6b-50ebf8290b0c/kube-rbac-proxy-web/0.log"
May 25 14:10:20.857855 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:10:20.857819    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_5f764e8d-9e15-478e-8e6b-50ebf8290b0c/kube-rbac-proxy/0.log"
May 25 14:10:20.889251 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:10:20.889218    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_5f764e8d-9e15-478e-8e6b-50ebf8290b0c/kube-rbac-proxy-metric/0.log"
May 25 14:10:20.919902 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:10:20.919869    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_5f764e8d-9e15-478e-8e6b-50ebf8290b0c/prom-label-proxy/0.log"
May 25 14:10:20.944032 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:10:20.944001    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_5f764e8d-9e15-478e-8e6b-50ebf8290b0c/init-config-reloader/0.log"
May 25 14:10:21.033413 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:10:21.033309    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-54fd6bc5c4-cns6h_f83b731b-63df-4555-95a3-ef82a12a049d/kube-state-metrics/0.log"
May 25 14:10:21.057761 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:10:21.057727    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-54fd6bc5c4-cns6h_f83b731b-63df-4555-95a3-ef82a12a049d/kube-rbac-proxy-main/0.log"
May 25 14:10:21.081245 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:10:21.081208    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-54fd6bc5c4-cns6h_f83b731b-63df-4555-95a3-ef82a12a049d/kube-rbac-proxy-self/0.log"
May 25 14:10:21.109673 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:10:21.109639    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-759cd6b97c-kmz7z_81e544fc-5a66-44c8-8301-0941bedeebe5/metrics-server/0.log"
May 25 14:10:21.135972 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:10:21.135945    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-665c9c8584-7s92b_5db702e4-41d1-42ec-b61d-f71b1cb2fe05/monitoring-plugin/0.log"
May 25 14:10:21.320132 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:10:21.320086    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-sxvw9_a6e33706-ba69-4aa3-9208-d2b97dfda444/node-exporter/0.log"
May 25 14:10:21.347070 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:10:21.347036    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-sxvw9_a6e33706-ba69-4aa3-9208-d2b97dfda444/kube-rbac-proxy/0.log"
May 25 14:10:21.370333 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:10:21.370293    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-sxvw9_a6e33706-ba69-4aa3-9208-d2b97dfda444/init-textfile/0.log"
May 25 14:10:21.401317 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:10:21.401289    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-9c959559f-klxgf_1bd034d7-2aef-4292-af7a-9a72e9fa621b/kube-rbac-proxy-main/0.log"
May 25 14:10:21.424934 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:10:21.424905    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-9c959559f-klxgf_1bd034d7-2aef-4292-af7a-9a72e9fa621b/kube-rbac-proxy-self/0.log"
May 25 14:10:21.454215 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:10:21.454185    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-9c959559f-klxgf_1bd034d7-2aef-4292-af7a-9a72e9fa621b/openshift-state-metrics/0.log"
May 25 14:10:21.528619 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:10:21.528565    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_5c8ede39-a1d6-41f5-9576-3fd866ec551d/prometheus/0.log"
May 25 14:10:21.546186 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:10:21.546149    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_5c8ede39-a1d6-41f5-9576-3fd866ec551d/config-reloader/0.log"
May 25 14:10:21.568034 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:10:21.567944    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_5c8ede39-a1d6-41f5-9576-3fd866ec551d/thanos-sidecar/0.log"
May 25 14:10:21.589219 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:10:21.589187    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_5c8ede39-a1d6-41f5-9576-3fd866ec551d/kube-rbac-proxy-web/0.log"
May 25 14:10:21.613998 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:10:21.613961    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_5c8ede39-a1d6-41f5-9576-3fd866ec551d/kube-rbac-proxy/0.log"
May 25 14:10:21.637525 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:10:21.637481    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_5c8ede39-a1d6-41f5-9576-3fd866ec551d/kube-rbac-proxy-thanos/0.log"
May 25 14:10:21.663920 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:10:21.663892    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_5c8ede39-a1d6-41f5-9576-3fd866ec551d/init-config-reloader/0.log"
May 25 14:10:21.701866 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:10:21.701816    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-7779bbdd78-vh2bs_72ab7f6c-5589-4ca4-88e4-e975cc094813/prometheus-operator/0.log"
May 25 14:10:21.721086 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:10:21.721053    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-7779bbdd78-vh2bs_72ab7f6c-5589-4ca4-88e4-e975cc094813/kube-rbac-proxy/0.log"
May 25 14:10:21.750262 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:10:21.750222    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-6c4bbc67b7-fnll9_91f38fd6-9e63-4273-8664-9f66efddcc5a/prometheus-operator-admission-webhook/0.log"
May 25 14:10:21.784449 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:10:21.784405    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6bb4767667-qjqdw_1e74c703-bb22-4ac6-8e96-1b429b819371/telemeter-client/0.log"
May 25 14:10:21.813559 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:10:21.813520    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6bb4767667-qjqdw_1e74c703-bb22-4ac6-8e96-1b429b819371/reload/0.log"
May 25 14:10:21.872494 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:10:21.872398    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6bb4767667-qjqdw_1e74c703-bb22-4ac6-8e96-1b429b819371/kube-rbac-proxy/0.log"
May 25 14:10:21.922833 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:10:21.922767    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-ddc675b94-jt4g7_1b20cb0e-7386-4292-860f-cae4aa862a94/thanos-query/0.log"
May 25 14:10:21.948030 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:10:21.947993    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-ddc675b94-jt4g7_1b20cb0e-7386-4292-860f-cae4aa862a94/kube-rbac-proxy-web/0.log"
May 25 14:10:21.978426 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:10:21.978342    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-ddc675b94-jt4g7_1b20cb0e-7386-4292-860f-cae4aa862a94/kube-rbac-proxy/0.log"
May 25 14:10:22.009600 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:10:22.009570    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-ddc675b94-jt4g7_1b20cb0e-7386-4292-860f-cae4aa862a94/prom-label-proxy/0.log"
May 25 14:10:22.033515 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:10:22.033482    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-ddc675b94-jt4g7_1b20cb0e-7386-4292-860f-cae4aa862a94/kube-rbac-proxy-rules/0.log"
May 25 14:10:22.061742 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:10:22.061707    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-ddc675b94-jt4g7_1b20cb0e-7386-4292-860f-cae4aa862a94/kube-rbac-proxy-metrics/0.log"
May 25 14:10:24.586115 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:10:24.586080    2557 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-must-gather-tl2jt/perf-node-gather-daemonset-qrwf7"]
May 25 14:10:24.594427 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:10:24.594395    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-tl2jt/perf-node-gather-daemonset-qrwf7"
May 25 14:10:24.600682 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:10:24.600650    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-tl2jt/perf-node-gather-daemonset-qrwf7"]
May 25 14:10:24.641107 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:10:24.641062    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c27221b7-271b-4f2c-86bc-5be78e6bc27d-podres\") pod \"perf-node-gather-daemonset-qrwf7\" (UID: \"c27221b7-271b-4f2c-86bc-5be78e6bc27d\") " pod="openshift-must-gather-tl2jt/perf-node-gather-daemonset-qrwf7"
May 25 14:10:24.641286 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:10:24.641113    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwhh5\" (UniqueName: \"kubernetes.io/projected/c27221b7-271b-4f2c-86bc-5be78e6bc27d-kube-api-access-jwhh5\") pod \"perf-node-gather-daemonset-qrwf7\" (UID: \"c27221b7-271b-4f2c-86bc-5be78e6bc27d\") " pod="openshift-must-gather-tl2jt/perf-node-gather-daemonset-qrwf7"
May 25 14:10:24.641286 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:10:24.641230    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c27221b7-271b-4f2c-86bc-5be78e6bc27d-proc\") pod \"perf-node-gather-daemonset-qrwf7\" (UID: \"c27221b7-271b-4f2c-86bc-5be78e6bc27d\") " pod="openshift-must-gather-tl2jt/perf-node-gather-daemonset-qrwf7"
May 25 14:10:24.641413 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:10:24.641284    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c27221b7-271b-4f2c-86bc-5be78e6bc27d-lib-modules\") pod \"perf-node-gather-daemonset-qrwf7\" (UID: \"c27221b7-271b-4f2c-86bc-5be78e6bc27d\") " pod="openshift-must-gather-tl2jt/perf-node-gather-daemonset-qrwf7"
May 25 14:10:24.641413 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:10:24.641403    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c27221b7-271b-4f2c-86bc-5be78e6bc27d-sys\") pod \"perf-node-gather-daemonset-qrwf7\" (UID: \"c27221b7-271b-4f2c-86bc-5be78e6bc27d\") " pod="openshift-must-gather-tl2jt/perf-node-gather-daemonset-qrwf7"
May 25 14:10:24.742774 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:10:24.742736    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c27221b7-271b-4f2c-86bc-5be78e6bc27d-proc\") pod \"perf-node-gather-daemonset-qrwf7\" (UID: \"c27221b7-271b-4f2c-86bc-5be78e6bc27d\") " pod="openshift-must-gather-tl2jt/perf-node-gather-daemonset-qrwf7"
May 25 14:10:24.742774 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:10:24.742779    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c27221b7-271b-4f2c-86bc-5be78e6bc27d-lib-modules\") pod \"perf-node-gather-daemonset-qrwf7\" (UID: \"c27221b7-271b-4f2c-86bc-5be78e6bc27d\") " pod="openshift-must-gather-tl2jt/perf-node-gather-daemonset-qrwf7"
May 25 14:10:24.742989 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:10:24.742820    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c27221b7-271b-4f2c-86bc-5be78e6bc27d-sys\") pod \"perf-node-gather-daemonset-qrwf7\" (UID: \"c27221b7-271b-4f2c-86bc-5be78e6bc27d\") " pod="openshift-must-gather-tl2jt/perf-node-gather-daemonset-qrwf7"
May 25 14:10:24.742989 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:10:24.742861    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c27221b7-271b-4f2c-86bc-5be78e6bc27d-proc\") pod \"perf-node-gather-daemonset-qrwf7\" (UID: \"c27221b7-271b-4f2c-86bc-5be78e6bc27d\") " pod="openshift-must-gather-tl2jt/perf-node-gather-daemonset-qrwf7"
May 25 14:10:24.742989 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:10:24.742870    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c27221b7-271b-4f2c-86bc-5be78e6bc27d-podres\") pod \"perf-node-gather-daemonset-qrwf7\" (UID: \"c27221b7-271b-4f2c-86bc-5be78e6bc27d\") " pod="openshift-must-gather-tl2jt/perf-node-gather-daemonset-qrwf7"
May 25 14:10:24.742989 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:10:24.742926    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c27221b7-271b-4f2c-86bc-5be78e6bc27d-sys\") pod \"perf-node-gather-daemonset-qrwf7\" (UID: \"c27221b7-271b-4f2c-86bc-5be78e6bc27d\") " pod="openshift-must-gather-tl2jt/perf-node-gather-daemonset-qrwf7"
May 25 14:10:24.742989 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:10:24.742938    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-jwhh5\" (UniqueName: \"kubernetes.io/projected/c27221b7-271b-4f2c-86bc-5be78e6bc27d-kube-api-access-jwhh5\") pod \"perf-node-gather-daemonset-qrwf7\" (UID: \"c27221b7-271b-4f2c-86bc-5be78e6bc27d\") " pod="openshift-must-gather-tl2jt/perf-node-gather-daemonset-qrwf7"
May 25 14:10:24.742989 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:10:24.742958    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c27221b7-271b-4f2c-86bc-5be78e6bc27d-podres\") pod \"perf-node-gather-daemonset-qrwf7\" (UID: \"c27221b7-271b-4f2c-86bc-5be78e6bc27d\") " pod="openshift-must-gather-tl2jt/perf-node-gather-daemonset-qrwf7"
May 25 14:10:24.742989 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:10:24.742964    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c27221b7-271b-4f2c-86bc-5be78e6bc27d-lib-modules\") pod \"perf-node-gather-daemonset-qrwf7\" (UID: \"c27221b7-271b-4f2c-86bc-5be78e6bc27d\") " pod="openshift-must-gather-tl2jt/perf-node-gather-daemonset-qrwf7"
May 25 14:10:24.751937 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:10:24.751897    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwhh5\" (UniqueName: \"kubernetes.io/projected/c27221b7-271b-4f2c-86bc-5be78e6bc27d-kube-api-access-jwhh5\") pod \"perf-node-gather-daemonset-qrwf7\" (UID: \"c27221b7-271b-4f2c-86bc-5be78e6bc27d\") " pod="openshift-must-gather-tl2jt/perf-node-gather-daemonset-qrwf7"
May 25 14:10:24.908284 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:10:24.908200    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-tl2jt/perf-node-gather-daemonset-qrwf7"
May 25 14:10:25.075865 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:10:25.075791    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-tl2jt/perf-node-gather-daemonset-qrwf7"]
May 25 14:10:25.078299 ip-10-0-143-235 kubenswrapper[2557]: W0525 14:10:25.078268    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podc27221b7_271b_4f2c_86bc_5be78e6bc27d.slice/crio-9bcb18c0311b0fcef7ce637bf0cb0d05587f4db89792e629f382a5d814478534 WatchSource:0}: Error finding container 9bcb18c0311b0fcef7ce637bf0cb0d05587f4db89792e629f382a5d814478534: Status 404 returned error can't find the container with id 9bcb18c0311b0fcef7ce637bf0cb0d05587f4db89792e629f382a5d814478534
May 25 14:10:25.145041 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:10:25.144838    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-tl2jt/perf-node-gather-daemonset-qrwf7" event={"ID":"c27221b7-271b-4f2c-86bc-5be78e6bc27d","Type":"ContainerStarted","Data":"9bcb18c0311b0fcef7ce637bf0cb0d05587f4db89792e629f382a5d814478534"}
May 25 14:10:25.643692 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:10:25.643664    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-5cwgw_776386fc-900b-494b-a190-cb092725bbf7/dns/0.log"
May 25 14:10:25.668261 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:10:25.668216    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-5cwgw_776386fc-900b-494b-a190-cb092725bbf7/kube-rbac-proxy/0.log"
May 25 14:10:25.825401 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:10:25.825375    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-hp2w9_50653f82-1790-4f22-94c1-406308112ca5/dns-node-resolver/0.log"
May 25 14:10:26.151559 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:10:26.151523    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-tl2jt/perf-node-gather-daemonset-qrwf7" event={"ID":"c27221b7-271b-4f2c-86bc-5be78e6bc27d","Type":"ContainerStarted","Data":"d919abc60d99af09024742e7f967d0dbe48f9051d2690e269c86daca3a30558b"}
May 25 14:10:26.151745 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:10:26.151675    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-tl2jt/perf-node-gather-daemonset-qrwf7"
May 25 14:10:26.179751 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:10:26.179700    2557 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-must-gather-tl2jt/perf-node-gather-daemonset-qrwf7" podStartSLOduration=2.1796819689999998 podStartE2EDuration="2.179681969s" podCreationTimestamp="2026-05-25 14:10:24 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 14:10:26.177198591 +0000 UTC m=+3878.640214752" watchObservedRunningTime="2026-05-25 14:10:26.179681969 +0000 UTC m=+3878.642698125"
May 25 14:10:26.431335 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:10:26.431259    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-v9wvs_9d1a5d53-d761-478d-a85e-0974f65121e3/node-ca/0.log"
May 25 14:10:27.838569 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:10:27.838542    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-5thz9_b8ace320-b067-48f1-9a9d-9f0681b3ee25/serve-healthcheck-canary/0.log"
May 25 14:10:28.387824 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:10:28.387799    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-glgst_4f20140e-81a7-4c76-9daa-eaa7057ba3d3/kube-rbac-proxy/0.log"
May 25 14:10:28.412647 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:10:28.412621    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-glgst_4f20140e-81a7-4c76-9daa-eaa7057ba3d3/exporter/0.log"
May 25 14:10:28.437117 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:10:28.437087    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-glgst_4f20140e-81a7-4c76-9daa-eaa7057ba3d3/extractor/0.log"
May 25 14:10:31.219770 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:10:31.219740    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-6f6cfd7d7-s6cd2_b920cc63-c662-4048-a720-68be65daa919/manager/0.log"
May 25 14:10:31.806969 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:10:31.806941    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_kserve-controller-manager-7cd9b78f5d-b6kcm_fd65f3b4-c9f6-4141-b034-92cf6bafa8f3/manager/0.log"
May 25 14:10:31.873412 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:10:31.873385    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_model-serving-api-86f7b4b499-kkplx_ee57af8c-4405-4ce5-8ed2-3c86dd820474/server/0.log"
May 25 14:10:32.169677 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:10:32.169592    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-tl2jt/perf-node-gather-daemonset-qrwf7"
May 25 14:10:38.567993 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:10:38.567960    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-lphh6_1dcac962-5867-4644-b3b3-ce62b3216b79/kube-multus-additional-cni-plugins/0.log"
May 25 14:10:38.588227 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:10:38.588198    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-lphh6_1dcac962-5867-4644-b3b3-ce62b3216b79/egress-router-binary-copy/0.log"
May 25 14:10:38.608294 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:10:38.608259    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-lphh6_1dcac962-5867-4644-b3b3-ce62b3216b79/cni-plugins/0.log"
May 25 14:10:38.635159 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:10:38.635127    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-lphh6_1dcac962-5867-4644-b3b3-ce62b3216b79/bond-cni-plugin/0.log"
May 25 14:10:38.655126 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:10:38.655097    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-lphh6_1dcac962-5867-4644-b3b3-ce62b3216b79/routeoverride-cni/0.log"
May 25 14:10:38.676256 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:10:38.676230    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-lphh6_1dcac962-5867-4644-b3b3-ce62b3216b79/whereabouts-cni-bincopy/0.log"
May 25 14:10:38.695918 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:10:38.695891    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-lphh6_1dcac962-5867-4644-b3b3-ce62b3216b79/whereabouts-cni/0.log"
May 25 14:10:38.969854 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:10:38.969777    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-scw67_87942b76-e2a7-407e-9a99-2ea92f737157/kube-multus/0.log"
May 25 14:10:39.021707 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:10:39.021674    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-5lm4j_8c2f1987-8965-449c-b426-b46515332392/network-metrics-daemon/0.log"
May 25 14:10:39.041162 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:10:39.041138    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-5lm4j_8c2f1987-8965-449c-b426-b46515332392/kube-rbac-proxy/0.log"
May 25 14:10:40.359326 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:10:40.359281    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vqnvc_d21bdf8b-d0ab-48f2-8736-54e92947dd53/ovn-controller/0.log"
May 25 14:10:40.389095 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:10:40.389069    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vqnvc_d21bdf8b-d0ab-48f2-8736-54e92947dd53/ovn-acl-logging/0.log"
May 25 14:10:40.406585 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:10:40.406540    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vqnvc_d21bdf8b-d0ab-48f2-8736-54e92947dd53/ovn-acl-logging/1.log"
May 25 14:10:40.426293 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:10:40.426265    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vqnvc_d21bdf8b-d0ab-48f2-8736-54e92947dd53/kube-rbac-proxy-node/0.log"
May 25 14:10:40.448570 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:10:40.448543    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vqnvc_d21bdf8b-d0ab-48f2-8736-54e92947dd53/kube-rbac-proxy-ovn-metrics/0.log"
May 25 14:10:40.471171 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:10:40.471141    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vqnvc_d21bdf8b-d0ab-48f2-8736-54e92947dd53/northd/0.log"
May 25 14:10:40.492686 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:10:40.492657    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vqnvc_d21bdf8b-d0ab-48f2-8736-54e92947dd53/nbdb/0.log"
May 25 14:10:40.514075 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:10:40.514043    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vqnvc_d21bdf8b-d0ab-48f2-8736-54e92947dd53/sbdb/0.log"
May 25 14:10:40.633549 ip-10-0-143-235 kubenswrapper[2557]: I0525 14:10:40.633474    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vqnvc_d21bdf8b-d0ab-48f2-8736-54e92947dd53/ovnkube-controller/0.log"