May 25 11:15:09.846938 ip-10-0-132-254 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
May 25 11:15:09.846947 ip-10-0-132-254 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
May 25 11:15:09.846954 ip-10-0-132-254 systemd[1]: kubelet.service: Failed with result 'resources'.
May 25 11:15:09.847200 ip-10-0-132-254 systemd[1]: Failed to start Kubernetes Kubelet.
May 25 11:15:19.057398 ip-10-0-132-254 systemd[1]: kubelet.service: Failed to schedule restart job: Unit openvswitch.service not found.
May 25 11:15:19.057411 ip-10-0-132-254 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot cddbd296494d44ebac740fe2b20d1673 --
May 25 11:17:34.351778 ip-10-0-132-254 systemd[1]: Starting Kubernetes Kubelet...
May 25 11:17:34.856842 ip-10-0-132-254 kubenswrapper[2576]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
May 25 11:17:34.856842 ip-10-0-132-254 kubenswrapper[2576]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
May 25 11:17:34.856842 ip-10-0-132-254 kubenswrapper[2576]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
May 25 11:17:34.856842 ip-10-0-132-254 kubenswrapper[2576]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
May 25 11:17:34.856842 ip-10-0-132-254 kubenswrapper[2576]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
May 25 11:17:34.858589 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:34.858496    2576 server.go:212] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
May 25 11:17:34.863688 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.863674    2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 25 11:17:34.863688 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.863687    2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 25 11:17:34.863750 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.863692    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 25 11:17:34.863750 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.863696    2576 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 25 11:17:34.863750 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.863699    2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 25 11:17:34.863750 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.863702    2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 25 11:17:34.863750 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.863705    2576 feature_gate.go:328] unrecognized feature gate: Example2
May 25 11:17:34.863750 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.863708    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 25 11:17:34.863750 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.863711    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 25 11:17:34.863750 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.863714    2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 25 11:17:34.863750 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.863717    2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 25 11:17:34.863750 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.863719    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 25 11:17:34.863750 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.863723    2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 25 11:17:34.863750 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.863725    2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 25 11:17:34.863750 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.863730    2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 25 11:17:34.863750 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.863739    2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 25 11:17:34.863750 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.863743    2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 25 11:17:34.863750 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.863746    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 25 11:17:34.863750 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.863749    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 25 11:17:34.863750 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.863753    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 25 11:17:34.863750 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.863756    2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 25 11:17:34.864207 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.863758    2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 25 11:17:34.864207 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.863761    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 25 11:17:34.864207 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.863764    2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 25 11:17:34.864207 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.863767    2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 25 11:17:34.864207 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.863770    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 25 11:17:34.864207 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.863773    2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 25 11:17:34.864207 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.863776    2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 25 11:17:34.864207 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.863778    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 25 11:17:34.864207 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.863781    2576 feature_gate.go:328] unrecognized feature gate: NewOLM
May 25 11:17:34.864207 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.863785    2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 25 11:17:34.864207 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.863789    2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 25 11:17:34.864207 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.863791    2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 25 11:17:34.864207 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.863794    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 25 11:17:34.864207 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.863796    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 25 11:17:34.864207 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.863799    2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 25 11:17:34.864207 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.863802    2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 25 11:17:34.864207 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.863805    2576 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 25 11:17:34.864207 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.863807    2576 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 25 11:17:34.864207 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.863810    2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 25 11:17:34.864207 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.863812    2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 25 11:17:34.864838 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.863815    2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 25 11:17:34.864838 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.863817    2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 25 11:17:34.864838 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.863820    2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 25 11:17:34.864838 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.863823    2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 25 11:17:34.864838 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.863825    2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 25 11:17:34.864838 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.863828    2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 25 11:17:34.864838 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.863830    2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 25 11:17:34.864838 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.863833    2576 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 25 11:17:34.864838 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.863835    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 25 11:17:34.864838 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.863838    2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 25 11:17:34.864838 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.863841    2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 25 11:17:34.864838 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.863843    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 25 11:17:34.864838 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.863846    2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 25 11:17:34.864838 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.863850    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 25 11:17:34.864838 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.863853    2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 25 11:17:34.864838 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.863855    2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 25 11:17:34.864838 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.863858    2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 25 11:17:34.864838 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.863861    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 25 11:17:34.864838 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.863863    2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 25 11:17:34.865321 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.863865    2576 feature_gate.go:328] unrecognized feature gate: DualReplica
May 25 11:17:34.865321 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.863868    2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 25 11:17:34.865321 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.863871    2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 25 11:17:34.865321 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.863874    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 25 11:17:34.865321 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.863877    2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 25 11:17:34.865321 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.863880    2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 25 11:17:34.865321 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.863888    2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 25 11:17:34.865321 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.863890    2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 25 11:17:34.865321 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.863893    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 25 11:17:34.865321 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.863896    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 25 11:17:34.865321 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.863898    2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 25 11:17:34.865321 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.863901    2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 25 11:17:34.865321 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.863903    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 25 11:17:34.865321 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.863906    2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 25 11:17:34.865321 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.863908    2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 25 11:17:34.865321 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.863911    2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 25 11:17:34.865321 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.863913    2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 25 11:17:34.865321 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.863916    2576 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 25 11:17:34.865321 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.863918    2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 25 11:17:34.865321 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.863921    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 25 11:17:34.865802 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.863923    2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 25 11:17:34.865802 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.863926    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 25 11:17:34.865802 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.863928    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 25 11:17:34.865802 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.863931    2576 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 25 11:17:34.865802 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.863933    2576 feature_gate.go:328] unrecognized feature gate: Example
May 25 11:17:34.865802 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.863936    2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 25 11:17:34.865802 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.864361    2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 25 11:17:34.865802 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.864366    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 25 11:17:34.865802 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.864369    2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 25 11:17:34.865802 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.864372    2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 25 11:17:34.865802 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.864375    2576 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 25 11:17:34.865802 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.864378    2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 25 11:17:34.865802 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.864381    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 25 11:17:34.865802 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.864383    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 25 11:17:34.865802 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.864386    2576 feature_gate.go:328] unrecognized feature gate: Example2
May 25 11:17:34.865802 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.864388    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 25 11:17:34.865802 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.864391    2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 25 11:17:34.865802 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.864394    2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 25 11:17:34.865802 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.864400    2576 feature_gate.go:328] unrecognized feature gate: Example
May 25 11:17:34.865802 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.864403    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 25 11:17:34.866298 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.864405    2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 25 11:17:34.866298 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.864408    2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 25 11:17:34.866298 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.864410    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 25 11:17:34.866298 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.864413    2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 25 11:17:34.866298 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.864416    2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 25 11:17:34.866298 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.864418    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 25 11:17:34.866298 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.864421    2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 25 11:17:34.866298 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.864424    2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 25 11:17:34.866298 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.864426    2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 25 11:17:34.866298 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.864429    2576 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 25 11:17:34.866298 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.864431    2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 25 11:17:34.866298 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.864434    2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 25 11:17:34.866298 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.864436    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 25 11:17:34.866298 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.864439    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 25 11:17:34.866298 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.864441    2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 25 11:17:34.866298 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.864444    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 25 11:17:34.866298 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.864447    2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 25 11:17:34.866298 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.864449    2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 25 11:17:34.866298 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.864452    2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 25 11:17:34.866298 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.864455    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 25 11:17:34.866794 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.864458    2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 25 11:17:34.866794 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.864461    2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 25 11:17:34.866794 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.864464    2576 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 25 11:17:34.866794 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.864466    2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 25 11:17:34.866794 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.864469    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 25 11:17:34.866794 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.864471    2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 25 11:17:34.866794 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.864474    2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 25 11:17:34.866794 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.864477    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 25 11:17:34.866794 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.864479    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 25 11:17:34.866794 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.864482    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 25 11:17:34.866794 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.864484    2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 25 11:17:34.866794 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.864487    2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 25 11:17:34.866794 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.864490    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 25 11:17:34.866794 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.864492    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 25 11:17:34.866794 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.864495    2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 25 11:17:34.866794 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.864497    2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 25 11:17:34.866794 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.864500    2576 feature_gate.go:328] unrecognized feature gate: DualReplica
May 25 11:17:34.866794 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.864502    2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 25 11:17:34.866794 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.864506    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 25 11:17:34.867258 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.864508    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 25 11:17:34.867258 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.864510    2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 25 11:17:34.867258 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.864513    2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 25 11:17:34.867258 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.864515    2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 25 11:17:34.867258 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.864518    2576 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 25 11:17:34.867258 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.864520    2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 25 11:17:34.867258 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.864523    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 25 11:17:34.867258 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.864525    2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 25 11:17:34.867258 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.864528    2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 25 11:17:34.867258 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.864532    2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 25 11:17:34.867258 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.864536    2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 25 11:17:34.867258 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.864539    2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 25 11:17:34.867258 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.864542    2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 25 11:17:34.867258 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.864544    2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 25 11:17:34.867258 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.864547    2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 25 11:17:34.867258 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.864549    2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 25 11:17:34.867258 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.864552    2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 25 11:17:34.867258 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.864555    2576 feature_gate.go:328] unrecognized feature gate: NewOLM
May 25 11:17:34.867258 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.864558    2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 25 11:17:34.867258 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.864560    2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 25 11:17:34.867760 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.864563    2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 25 11:17:34.867760 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.864565    2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 25 11:17:34.867760 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.864567    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 25 11:17:34.867760 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.864570    2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 25 11:17:34.867760 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.864573    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 25 11:17:34.867760 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.864576    2576 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 25 11:17:34.867760 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.864579    2576 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 25 11:17:34.867760 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.864581    2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 25 11:17:34.867760 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.864584    2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 25 11:17:34.867760 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.864587    2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 25 11:17:34.867760 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.864589    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 25 11:17:34.867760 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.864593    2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 25 11:17:34.867760 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.864596    2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 25 11:17:34.867760 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:34.866374    2576 flags.go:64] FLAG: --address="0.0.0.0"
May 25 11:17:34.867760 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:34.866388    2576 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
May 25 11:17:34.867760 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:34.866396    2576 flags.go:64] FLAG: --anonymous-auth="true"
May 25 11:17:34.867760 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:34.866403    2576 flags.go:64] FLAG: --application-metrics-count-limit="100"
May 25 11:17:34.867760 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:34.866407    2576 flags.go:64] FLAG: --authentication-token-webhook="false"
May 25 11:17:34.867760 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:34.866410    2576 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
May 25 11:17:34.867760 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:34.866415    2576 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
May 25 11:17:34.867760 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:34.866420    2576 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
May 25 11:17:34.868288 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:34.866423    2576 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
May 25 11:17:34.868288 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:34.866426    2576 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
May 25 11:17:34.868288 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:34.866430    2576 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
May 25 11:17:34.868288 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:34.866434    2576 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
May 25 11:17:34.868288 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:34.866437    2576 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
May 25 11:17:34.868288 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:34.866440    2576 flags.go:64] FLAG: --cgroup-root=""
May 25 11:17:34.868288 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:34.866443    2576 flags.go:64] FLAG: --cgroups-per-qos="true"
May 25 11:17:34.868288 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:34.866446    2576 flags.go:64] FLAG: --client-ca-file=""
May 25 11:17:34.868288 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:34.866449    2576 flags.go:64] FLAG: --cloud-config=""
May 25 11:17:34.868288 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:34.866452    2576 flags.go:64] FLAG: --cloud-provider="external"
May 25 11:17:34.868288 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:34.866455    2576 flags.go:64] FLAG: --cluster-dns="[]"
May 25 11:17:34.868288 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:34.866460    2576 flags.go:64] FLAG: --cluster-domain=""
May 25 11:17:34.868288 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:34.866463    2576 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
May 25 11:17:34.868288 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:34.866466    2576 flags.go:64] FLAG: --config-dir=""
May 25 11:17:34.868288 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:34.866468    2576 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
May 25 11:17:34.868288 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:34.866472    2576 flags.go:64] FLAG: --container-log-max-files="5"
May 25 11:17:34.868288 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:34.866476    2576 flags.go:64] FLAG: --container-log-max-size="10Mi"
May 25 11:17:34.868288 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:34.866480    2576 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
May 25 11:17:34.868288 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:34.866483    2576 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
May 25 11:17:34.868288 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:34.866486    2576 flags.go:64] FLAG: --containerd-namespace="k8s.io"
May 25 11:17:34.868288 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:34.866489    2576 flags.go:64] FLAG: --contention-profiling="false"
May 25 11:17:34.868288 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:34.866492    2576 flags.go:64] FLAG: --cpu-cfs-quota="true"
May 25 11:17:34.868288 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:34.866495    2576 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
May 25 11:17:34.868288 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:34.866498    2576 flags.go:64] FLAG: --cpu-manager-policy="none"
May 25 11:17:34.868288 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:34.866501    2576 flags.go:64] FLAG: --cpu-manager-policy-options=""
May 25 11:17:34.868892 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:34.866505    2576 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
May 25 11:17:34.868892 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:34.866508    2576 flags.go:64] FLAG: --enable-controller-attach-detach="true"
May 25 11:17:34.868892 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:34.866511    2576 flags.go:64] FLAG: --enable-debugging-handlers="true"
May 25 11:17:34.868892 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:34.866513    2576 flags.go:64] FLAG: --enable-load-reader="false"
May 25 11:17:34.868892 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:34.866516    2576 flags.go:64] FLAG: --enable-server="true"
May 25 11:17:34.868892 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:34.866520    2576 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
May 25 11:17:34.868892 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:34.866524    2576 flags.go:64] FLAG: --event-burst="100"
May 25 11:17:34.868892 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:34.866527    2576 flags.go:64] FLAG: --event-qps="50"
May 25 11:17:34.868892 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:34.866530    2576 flags.go:64] FLAG: --event-storage-age-limit="default=0"
May 25 11:17:34.868892 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:34.866534    2576 flags.go:64] FLAG: --event-storage-event-limit="default=0"
May 25 11:17:34.868892 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:34.866537    2576 flags.go:64] FLAG: --eviction-hard=""
May 25 11:17:34.868892 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:34.866540    2576 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
May 25 11:17:34.868892 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:34.866544    2576 flags.go:64] FLAG: --eviction-minimum-reclaim=""
May 25 11:17:34.868892 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:34.866547    2576 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
May 25 11:17:34.868892 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:34.866550    2576 flags.go:64] FLAG: --eviction-soft=""
May 25 11:17:34.868892 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:34.866553    2576 flags.go:64] FLAG: --eviction-soft-grace-period=""
May 25 11:17:34.868892 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:34.866556    2576 flags.go:64] FLAG: --exit-on-lock-contention="false"
May 25 11:17:34.868892 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:34.866559    2576 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
May 25 11:17:34.868892 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:34.866562    2576 flags.go:64] FLAG: --experimental-mounter-path=""
May 25 11:17:34.868892 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:34.866565    2576 flags.go:64] FLAG: --fail-cgroupv1="false"
May 25 11:17:34.868892 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:34.866568    2576 flags.go:64] FLAG: --fail-swap-on="true"
May 25 11:17:34.868892 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:34.866571    2576 flags.go:64] FLAG: --feature-gates=""
May 25 11:17:34.868892 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:34.866575    2576 flags.go:64] FLAG: --file-check-frequency="20s"
May 25 11:17:34.868892 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:34.866578    2576 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
May 25 11:17:34.868892 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:34.866581    2576 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
May 25 11:17:34.869512 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:34.866585    2576 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
May 25 11:17:34.869512 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:34.866589    2576 flags.go:64] FLAG: --healthz-port="10248"
May 25 11:17:34.869512 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:34.866592    2576 flags.go:64] FLAG: --help="false"
May 25 11:17:34.869512 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:34.866595    2576 flags.go:64] FLAG: --hostname-override="ip-10-0-132-254.ec2.internal"
May 25 11:17:34.869512 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:34.866598    2576 flags.go:64] FLAG: --housekeeping-interval="10s"
May 25 11:17:34.869512 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:34.866601    2576 flags.go:64] FLAG: --http-check-frequency="20s"
May 25 11:17:34.869512 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:34.866604    2576 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
May 25 11:17:34.869512 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:34.866607    2576 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
May 25 11:17:34.869512 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:34.866611    2576 flags.go:64] FLAG: --image-gc-high-threshold="85"
May 25 11:17:34.869512 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:34.866614    2576 flags.go:64] FLAG: --image-gc-low-threshold="80"
May 25 11:17:34.869512 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:34.866616    2576 flags.go:64] FLAG: --image-service-endpoint=""
May 25 11:17:34.869512 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:34.866619    2576 flags.go:64] FLAG: --kernel-memcg-notification="false"
May 25 11:17:34.869512 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:34.866622    2576 flags.go:64] FLAG: --kube-api-burst="100"
May 25 11:17:34.869512 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:34.866625    2576 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
May 25 11:17:34.869512 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:34.866629    2576 flags.go:64] FLAG: --kube-api-qps="50"
May 25 11:17:34.869512 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:34.866632    2576 flags.go:64] FLAG: --kube-reserved=""
May 25 11:17:34.869512 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:34.866635    2576 flags.go:64] FLAG: --kube-reserved-cgroup=""
May 25 11:17:34.869512 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:34.866638    2576 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
May 25 11:17:34.869512 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:34.866641    2576 flags.go:64] FLAG: --kubelet-cgroups=""
May 25 11:17:34.869512 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:34.866644    2576 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
May 25 11:17:34.869512 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:34.866647    2576 flags.go:64] FLAG: --lock-file=""
May 25 11:17:34.869512 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:34.866650    2576 flags.go:64] FLAG: --log-cadvisor-usage="false"
May 25 11:17:34.869512 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:34.866653    2576 flags.go:64] FLAG: --log-flush-frequency="5s"
May 25 11:17:34.869512 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:34.866656    2576 flags.go:64] FLAG: --log-json-info-buffer-size="0"
May 25 11:17:34.870138 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:34.866661    2576 flags.go:64] FLAG: --log-json-split-stream="false"
May 25 11:17:34.870138 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:34.866664    2576 flags.go:64] FLAG: --log-text-info-buffer-size="0"
May 25 11:17:34.870138 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:34.866667    2576 flags.go:64] FLAG: --log-text-split-stream="false"
May 25 11:17:34.870138 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:34.866670    2576 flags.go:64] FLAG: --logging-format="text"
May 25 11:17:34.870138 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:34.866673    2576 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
May 25 11:17:34.870138 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:34.866676    2576 flags.go:64] FLAG: --make-iptables-util-chains="true"
May 25 11:17:34.870138 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:34.866679    2576 flags.go:64] FLAG: --manifest-url=""
May 25 11:17:34.870138 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:34.866682    2576 flags.go:64] FLAG: --manifest-url-header=""
May 25 11:17:34.870138 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:34.866687    2576 flags.go:64] FLAG: --max-housekeeping-interval="15s"
May 25 11:17:34.870138 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:34.866690    2576 flags.go:64] FLAG: --max-open-files="1000000"
May 25 11:17:34.870138 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:34.866694    2576 flags.go:64] FLAG: --max-pods="110"
May 25 11:17:34.870138 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:34.866697    2576 flags.go:64] FLAG: --maximum-dead-containers="-1"
May 25 11:17:34.870138 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:34.866701    2576 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
May 25 11:17:34.870138 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:34.866704    2576 flags.go:64] FLAG: --memory-manager-policy="None"
May 25 11:17:34.870138 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:34.866706    2576 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
May 25 11:17:34.870138 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:34.866710    2576 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
May 25 11:17:34.870138 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:34.866712    2576 flags.go:64] FLAG: --node-ip="0.0.0.0"
May 25 11:17:34.870138 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:34.866715    2576 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
May 25 11:17:34.870138 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:34.866723    2576 flags.go:64] FLAG: --node-status-max-images="50"
May 25 11:17:34.870138 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:34.866726    2576 flags.go:64] FLAG: --node-status-update-frequency="10s"
May 25 11:17:34.870138 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:34.866730    2576 flags.go:64] FLAG: --oom-score-adj="-999"
May 25 11:17:34.870138 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:34.866733    2576 flags.go:64] FLAG: --pod-cidr=""
May 25 11:17:34.870138 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:34.866736    2576 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:afe88671bf163c42b467d159dc6f0b784dbe28b1ed44e171edc9c06fd99cc6fc"
May 25 11:17:34.870716 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:34.866742    2576 flags.go:64] FLAG: --pod-manifest-path=""
May 25 11:17:34.870716 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:34.866745    2576 flags.go:64] FLAG: --pod-max-pids="-1"
May 25 11:17:34.870716 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:34.866748    2576 flags.go:64] FLAG: --pods-per-core="0"
May 25 11:17:34.870716 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:34.866751    2576 flags.go:64] FLAG: --port="10250"
May 25 11:17:34.870716 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:34.866754    2576 flags.go:64] FLAG: --protect-kernel-defaults="false"
May 25 11:17:34.870716 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:34.866757    2576 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0fc96abdef4ceb6f5"
May 25 11:17:34.870716 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:34.866760    2576 flags.go:64] FLAG: --qos-reserved=""
May 25 11:17:34.870716 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:34.866763    2576 flags.go:64] FLAG: --read-only-port="10255"
May 25 11:17:34.870716 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:34.866766    2576 flags.go:64] FLAG: --register-node="true"
May 25 11:17:34.870716 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:34.866769    2576 flags.go:64] FLAG: --register-schedulable="true"
May 25 11:17:34.870716 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:34.866771    2576 flags.go:64] FLAG: --register-with-taints=""
May 25 11:17:34.870716 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:34.866775    2576 flags.go:64] FLAG: --registry-burst="10"
May 25 11:17:34.870716 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:34.866778    2576 flags.go:64] FLAG: --registry-qps="5"
May 25 11:17:34.870716 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:34.866781    2576 flags.go:64] FLAG: --reserved-cpus=""
May 25 11:17:34.870716 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:34.866783    2576 flags.go:64] FLAG: --reserved-memory=""
May 25 11:17:34.870716 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:34.866787    2576 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
May 25 11:17:34.870716 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:34.866790    2576 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
May 25 11:17:34.870716 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:34.866793    2576 flags.go:64] FLAG: --rotate-certificates="false"
May 25 11:17:34.870716 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:34.866796    2576 flags.go:64] FLAG: --rotate-server-certificates="false"
May 25 11:17:34.870716 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:34.866800    2576 flags.go:64] FLAG: --runonce="false"
May 25 11:17:34.870716 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:34.866803    2576 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
May 25 11:17:34.870716 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:34.866806    2576 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
May 25 11:17:34.870716 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:34.866810    2576 flags.go:64] FLAG: --seccomp-default="false"
May 25 11:17:34.870716 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:34.866813    2576 flags.go:64] FLAG: --serialize-image-pulls="true"
May 25 11:17:34.870716 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:34.866815    2576 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
May 25 11:17:34.870716 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:34.866818    2576 flags.go:64] FLAG: --storage-driver-db="cadvisor"
May 25 11:17:34.871382 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:34.866822    2576 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
May 25 11:17:34.871382 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:34.866825    2576 flags.go:64] FLAG: --storage-driver-password="root"
May 25 11:17:34.871382 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:34.866828    2576 flags.go:64] FLAG: --storage-driver-secure="false"
May 25 11:17:34.871382 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:34.866831    2576 flags.go:64] FLAG: --storage-driver-table="stats"
May 25 11:17:34.871382 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:34.866834    2576 flags.go:64] FLAG: --storage-driver-user="root"
May 25 11:17:34.871382 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:34.866837    2576 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
May 25 11:17:34.871382 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:34.866840    2576 flags.go:64] FLAG: --sync-frequency="1m0s"
May 25 11:17:34.871382 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:34.866843    2576 flags.go:64] FLAG: --system-cgroups=""
May 25 11:17:34.871382 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:34.866849    2576 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
May 25 11:17:34.871382 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:34.866855    2576 flags.go:64] FLAG: --system-reserved-cgroup=""
May 25 11:17:34.871382 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:34.866858    2576 flags.go:64] FLAG: --tls-cert-file=""
May 25 11:17:34.871382 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:34.866861    2576 flags.go:64] FLAG: --tls-cipher-suites="[]"
May 25 11:17:34.871382 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:34.866865    2576 flags.go:64] FLAG: --tls-min-version=""
May 25 11:17:34.871382 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:34.866868    2576 flags.go:64] FLAG: --tls-private-key-file=""
May 25 11:17:34.871382 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:34.866871    2576 flags.go:64] FLAG: --topology-manager-policy="none"
May 25 11:17:34.871382 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:34.866874    2576 flags.go:64] FLAG: --topology-manager-policy-options=""
May 25 11:17:34.871382 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:34.866877    2576 flags.go:64] FLAG: --topology-manager-scope="container"
May 25 11:17:34.871382 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:34.866880    2576 flags.go:64] FLAG: --v="2"
May 25 11:17:34.871382 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:34.866885    2576 flags.go:64] FLAG: --version="false"
May 25 11:17:34.871382 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:34.866889    2576 flags.go:64] FLAG: --vmodule=""
May 25 11:17:34.871382 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:34.866894    2576 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
May 25 11:17:34.871382 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:34.866897    2576 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
May 25 11:17:34.871382 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.867004    2576 feature_gate.go:328] unrecognized feature gate: DualReplica
May 25 11:17:34.871382 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.867009    2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 25 11:17:34.871382 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.867012    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 25 11:17:34.871966 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.867015    2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 25 11:17:34.871966 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.867019    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 25 11:17:34.871966 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.867022    2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 25 11:17:34.871966 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.867024    2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 25 11:17:34.871966 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.867027    2576 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 25 11:17:34.871966 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.867031    2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 25 11:17:34.871966 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.867035    2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 25 11:17:34.871966 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.867038    2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 25 11:17:34.871966 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.867040    2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 25 11:17:34.871966 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.867043    2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 25 11:17:34.871966 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.867046    2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 25 11:17:34.871966 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.867051    2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 25 11:17:34.871966 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.867054    2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 25 11:17:34.871966 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.867057    2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 25 11:17:34.871966 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.867060    2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 25 11:17:34.871966 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.867065    2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 25 11:17:34.871966 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.867067    2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 25 11:17:34.871966 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.867070    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 25 11:17:34.871966 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.867073    2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 25 11:17:34.872451 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.867076    2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 25 11:17:34.872451 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.867078    2576 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 25 11:17:34.872451 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.867081    2576 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 25 11:17:34.872451 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.867083    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 25 11:17:34.872451 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.867087    2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 25 11:17:34.872451 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.867090    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 25 11:17:34.872451 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.867092    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 25 11:17:34.872451 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.867095    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 25 11:17:34.872451 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.867097    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 25 11:17:34.872451 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.867100    2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 25 11:17:34.872451 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.867102    2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 25 11:17:34.872451 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.867105    2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 25 11:17:34.872451 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.867108    2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 25 11:17:34.872451 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.867110    2576 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 25 11:17:34.872451 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.867114    2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 25 11:17:34.872451 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.867118    2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 25 11:17:34.872451 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.867121    2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 25 11:17:34.872451 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.867123    2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 25 11:17:34.872451 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.867126    2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 25 11:17:34.872451 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.867128    2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 25 11:17:34.872945 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.867131    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 25 11:17:34.872945 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.867134    2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 25 11:17:34.872945 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.867136    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 25 11:17:34.872945 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.867139    2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 25 11:17:34.872945 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.867141    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 25 11:17:34.872945 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.867144    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 25 11:17:34.872945 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.867147    2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 25 11:17:34.872945 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.867149    2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 25 11:17:34.872945 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.867153    2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 25 11:17:34.872945 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.867156    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 25 11:17:34.872945 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.867159    2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 25 11:17:34.872945 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.867161    2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 25 11:17:34.872945 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.867164    2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 25 11:17:34.872945 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.867167    2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 25 11:17:34.872945 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.867169    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 25 11:17:34.872945 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.867172    2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 25 11:17:34.872945 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.867175    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 25 11:17:34.872945 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.867178    2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 25 11:17:34.872945 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.867180    2576 feature_gate.go:328] unrecognized feature gate: Example2
May 25 11:17:34.872945 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.867183    2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 25 11:17:34.873510 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.867185    2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 25 11:17:34.873510 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.867188    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 25 11:17:34.873510 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.867190    2576 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 25 11:17:34.873510 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.867193    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 25 11:17:34.873510 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.867195    2576 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 25 11:17:34.873510 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.867198    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 25 11:17:34.873510 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.867205    2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 25 11:17:34.873510 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.867208    2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 25 11:17:34.873510 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.867211    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 25 11:17:34.873510 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.867214    2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 25 11:17:34.873510 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.867217    2576 feature_gate.go:328] unrecognized feature gate: NewOLM
May 25 11:17:34.873510 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.867219    2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 25 11:17:34.873510 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.867222    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 25 11:17:34.873510 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.867224    2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 25 11:17:34.873510 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.867227    2576 feature_gate.go:328] unrecognized feature gate: Example
May 25 11:17:34.873510 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.867230    2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 25 11:17:34.873510 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.867232    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 25 11:17:34.873510 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.867235    2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 25 11:17:34.873510 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.867237    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 25 11:17:34.873510 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.867240    2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 25 11:17:34.874006 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.867244    2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 25 11:17:34.874006 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.867247    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 25 11:17:34.874006 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.867249    2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 25 11:17:34.874006 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.867252    2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 25 11:17:34.874006 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:34.868060    2576 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]}
May 25 11:17:34.875652 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:34.875634    2576 server.go:530] "Kubelet version" kubeletVersion="v1.33.11"
May 25 11:17:34.875693 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:34.875654    2576 server.go:532] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
May 25 11:17:34.875723 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.875702    2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 25 11:17:34.875723 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.875707    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 25 11:17:34.875723 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.875711    2576 feature_gate.go:328] unrecognized feature gate: DualReplica
May 25 11:17:34.875723 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.875714    2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 25 11:17:34.875723 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.875717    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 25 11:17:34.875723 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.875721    2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 25 11:17:34.875723 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.875724    2576 feature_gate.go:328] unrecognized feature gate: NewOLM
May 25 11:17:34.875892 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.875727    2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 25 11:17:34.875892 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.875731    2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 25 11:17:34.875892 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.875734    2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 25 11:17:34.875892 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.875736    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 25 11:17:34.875892 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.875739    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 25 11:17:34.875892 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.875742    2576 feature_gate.go:328] unrecognized feature gate: Example
May 25 11:17:34.875892 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.875745    2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 25 11:17:34.875892 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.875748    2576 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 25 11:17:34.875892 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.875750    2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 25 11:17:34.875892 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.875753    2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 25 11:17:34.875892 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.875756    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 25 11:17:34.875892 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.875758    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 25 11:17:34.875892 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.875761    2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 25 11:17:34.875892 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.875763    2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 25 11:17:34.875892 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.875766    2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 25 11:17:34.875892 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.875768    2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 25 11:17:34.875892 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.875771    2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 25 11:17:34.875892 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.875773    2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 25 11:17:34.875892 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.875777    2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 25 11:17:34.875892 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.875779    2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 25 11:17:34.876474 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.875782    2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 25 11:17:34.876474 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.875784    2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 25 11:17:34.876474 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.875787    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 25 11:17:34.876474 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.875789    2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 25 11:17:34.876474 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.875793    2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 25 11:17:34.876474 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.875796    2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 25 11:17:34.876474 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.875798    2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 25 11:17:34.876474 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.875801    2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 25 11:17:34.876474 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.875803    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 25 11:17:34.876474 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.875806    2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 25 11:17:34.876474 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.875809    2576 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 25 11:17:34.876474 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.875811    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 25 11:17:34.876474 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.875814    2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 25 11:17:34.876474 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.875817    2576 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 25 11:17:34.876474 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.875820    2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 25 11:17:34.876474 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.875823    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 25 11:17:34.876474 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.875825    2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 25 11:17:34.876474 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.875828    2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 25 11:17:34.876474 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.875830    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 25 11:17:34.876474 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.875833    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 25 11:17:34.876957 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.875836    2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 25 11:17:34.876957 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.875838    2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 25 11:17:34.876957 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.875841    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 25 11:17:34.876957 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.875843    2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 25 11:17:34.876957 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.875846    2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 25 11:17:34.876957 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.875849    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 25 11:17:34.876957 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.875852    2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 25 11:17:34.876957 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.875856    2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 25 11:17:34.876957 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.875859    2576 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 25 11:17:34.876957 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.875863    2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 25 11:17:34.876957 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.875867    2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 25 11:17:34.876957 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.875871    2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 25 11:17:34.876957 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.875874    2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 25 11:17:34.876957 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.875877    2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 25 11:17:34.876957 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.875879    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 25 11:17:34.876957 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.875882    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 25 11:17:34.876957 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.875885    2576 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 25 11:17:34.876957 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.875888    2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 25 11:17:34.876957 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.875891    2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 25 11:17:34.877429 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.875894    2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 25 11:17:34.877429 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.875897    2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 25 11:17:34.877429 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.875900    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 25 11:17:34.877429 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.875902    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 25 11:17:34.877429 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.875905    2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 25 11:17:34.877429 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.875908    2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 25 11:17:34.877429 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.875912    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 25 11:17:34.877429 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.875915    2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 25 11:17:34.877429 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.875918    2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 25 11:17:34.877429 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.875921    2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 25 11:17:34.877429 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.875923    2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 25 11:17:34.877429 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.875926    2576 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 25 11:17:34.877429 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.875929    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 25 11:17:34.877429 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.875931    2576 feature_gate.go:328] unrecognized feature gate: Example2
May 25 11:17:34.877429 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.875934    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 25 11:17:34.877429 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.875936    2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 25 11:17:34.877429 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.875939    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 25 11:17:34.877429 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.875941    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 25 11:17:34.877429 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.875944    2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 25 11:17:34.877429 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.875946    2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 25 11:17:34.877901 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:34.875951    2576 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]}
May 25 11:17:34.877901 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.876048    2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 25 11:17:34.877901 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.876054    2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 25 11:17:34.877901 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.876057    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 25 11:17:34.877901 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.876060    2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 25 11:17:34.877901 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.876063    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 25 11:17:34.877901 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.876066    2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 25 11:17:34.877901 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.876069    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 25 11:17:34.877901 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.876071    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 25 11:17:34.877901 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.876074    2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 25 11:17:34.877901 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.876078    2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 25 11:17:34.877901 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.876083    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 25 11:17:34.877901 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.876086    2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 25 11:17:34.877901 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.876089    2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 25 11:17:34.877901 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.876091    2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 25 11:17:34.878416 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.876094    2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 25 11:17:34.878416 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.876097    2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 25 11:17:34.878416 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.876100    2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 25 11:17:34.878416 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.876102    2576 feature_gate.go:328] unrecognized feature gate: Example2
May 25 11:17:34.878416 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.876105    2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 25 11:17:34.878416 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.876108    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 25 11:17:34.878416 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.876110    2576 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 25 11:17:34.878416 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.876113    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 25 11:17:34.878416 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.876116    2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 25 11:17:34.878416 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.876118    2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 25 11:17:34.878416 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.876121    2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 25 11:17:34.878416 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.876123    2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 25 11:17:34.878416 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.876126    2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 25 11:17:34.878416 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.876128    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 25 11:17:34.878416 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.876131    2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 25 11:17:34.878416 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.876133    2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 25 11:17:34.878416 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.876136    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 25 11:17:34.878416 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.876138    2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 25 11:17:34.878416 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.876141    2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 25 11:17:34.878416 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.876143    2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 25 11:17:34.879189 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.876146    2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 25 11:17:34.879189 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.876148    2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 25 11:17:34.879189 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.876151    2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 25 11:17:34.879189 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.876153    2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 25 11:17:34.879189 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.876156    2576 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 25 11:17:34.879189 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.876159    2576 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 25 11:17:34.879189 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.876161    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 25 11:17:34.879189 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.876164    2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 25 11:17:34.879189 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.876167    2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 25 11:17:34.879189 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.876170    2576 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 25 11:17:34.879189 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.876173    2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 25 11:17:34.879189 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.876175    2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 25 11:17:34.879189 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.876178    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 25 11:17:34.879189 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.876181    2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 25 11:17:34.879189 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.876183    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 25 11:17:34.879189 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.876186    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 25 11:17:34.879189 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.876189    2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 25 11:17:34.879189 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.876191    2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 25 11:17:34.879189 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.876194    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 25 11:17:34.879189 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.876196    2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 25 11:17:34.879712 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.876199    2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 25 11:17:34.879712 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.876201    2576 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 25 11:17:34.879712 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.876204    2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 25 11:17:34.879712 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.876206    2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 25 11:17:34.879712 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.876209    2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 25 11:17:34.879712 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.876211    2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 25 11:17:34.879712 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.876214    2576 feature_gate.go:328] unrecognized feature gate: Example
May 25 11:17:34.879712 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.876217    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 25 11:17:34.879712 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.876219    2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 25 11:17:34.879712 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.876221    2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 25 11:17:34.879712 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.876224    2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 25 11:17:34.879712 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.876226    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 25 11:17:34.879712 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.876230    2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 25 11:17:34.879712 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.876234    2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 25 11:17:34.879712 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.876236    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 25 11:17:34.879712 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.876239    2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 25 11:17:34.879712 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.876242    2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 25 11:17:34.879712 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.876244    2576 feature_gate.go:328] unrecognized feature gate: DualReplica
May 25 11:17:34.879712 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.876247    2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 25 11:17:34.880175 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.876249    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 25 11:17:34.880175 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.876252    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 25 11:17:34.880175 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.876254    2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 25 11:17:34.880175 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.876258    2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 25 11:17:34.880175 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.876260    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 25 11:17:34.880175 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.876263    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 25 11:17:34.880175 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.876265    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 25 11:17:34.880175 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.876268    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 25 11:17:34.880175 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.876270    2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 25 11:17:34.880175 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.876273    2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 25 11:17:34.880175 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.876288    2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 25 11:17:34.880175 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.876291    2576 feature_gate.go:328] unrecognized feature gate: NewOLM
May 25 11:17:34.880175 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:34.876294    2576 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 25 11:17:34.880175 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:34.876299    2576 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]}
May 25 11:17:34.880175 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:34.877186    2576 server.go:962] "Client rotation is on, will bootstrap in background"
May 25 11:17:34.882608 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:34.882591    2576 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
May 25 11:17:34.883761 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:34.883750    2576 server.go:1019] "Starting client certificate rotation"
May 25 11:17:34.883862 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:34.883847    2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
May 25 11:17:34.883898 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:34.883889    2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
May 25 11:17:34.912660 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:34.912641    2576 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
May 25 11:17:34.917896 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:34.917629    2576 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
May 25 11:17:34.934709 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:34.934684    2576 log.go:25] "Validated CRI v1 runtime API"
May 25 11:17:34.942361 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:34.942343    2576 log.go:25] "Validated CRI v1 image API"
May 25 11:17:34.944443 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:34.944421    2576 server.go:1452] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
May 25 11:17:34.946382 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:34.946361    2576 reflector.go:430] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
May 25 11:17:34.947094 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:34.947074    2576 fs.go:135] Filesystem UUIDs: map[78d1d54b-8bed-4afc-a5ab-44c15df0b3c6:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 d2bd3c37-90fc-46df-8112-c46109c2e5a5:/dev/nvme0n1p4]
May 25 11:17:34.947135 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:34.947094    2576 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
May 25 11:17:34.952033 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:34.951918    2576 manager.go:217] Machine: {Timestamp:2026-05-25 11:17:34.951394789 +0000 UTC m=+0.469126255 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3097483 MemoryCapacity:33164492800 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2e666d032c0217c93478cbaf8e8128 SystemUUID:ec2e666d-032c-0217-c934-78cbaf8e8128 BootID:cddbd296-494d-44eb-ac74-0fe2b20d1673 Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6103040 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:df:99:7b:31:c9 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:df:99:7b:31:c9 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:3a:26:47:7b:d2:d7 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164492800 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
May 25 11:17:34.952839 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:34.952828    2576 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
May 25 11:17:34.952935 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:34.952922    2576 manager.go:233] Version: {KernelVersion:5.14.0-570.114.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260512-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
May 25 11:17:34.956030 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:34.955999    2576 container_manager_linux.go:270] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
May 25 11:17:34.956175 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:34.956031    2576 container_manager_linux.go:275] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-132-254.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
May 25 11:17:34.956221 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:34.956185    2576 topology_manager.go:138] "Creating topology manager with none policy"
May 25 11:17:34.956221 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:34.956194    2576 container_manager_linux.go:306] "Creating device plugin manager"
May 25 11:17:34.956221 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:34.956206    2576 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
May 25 11:17:34.957033 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:34.957022    2576 server.go:72] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
May 25 11:17:34.959623 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:34.959611    2576 state_mem.go:36] "Initialized new in-memory state store"
May 25 11:17:34.959738 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:34.959729    2576 server.go:1267] "Using root directory" path="/var/lib/kubelet"
May 25 11:17:34.962385 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:34.962374    2576 kubelet.go:491] "Attempting to sync node with API server"
May 25 11:17:34.962424 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:34.962390    2576 kubelet.go:386] "Adding static pod path" path="/etc/kubernetes/manifests"
May 25 11:17:34.962424 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:34.962402    2576 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
May 25 11:17:34.962424 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:34.962412    2576 kubelet.go:397] "Adding apiserver pod source"
May 25 11:17:34.962542 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:34.962424    2576 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
May 25 11:17:34.966223 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:34.966195    2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
May 25 11:17:34.966362 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:34.966241    2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
May 25 11:17:34.970413 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:34.970394    2576 kuberuntime_manager.go:279] "Container runtime initialized" containerRuntime="cri-o" version="1.33.11-2.rhaos4.20.gitb2a8320.el9" apiVersion="v1"
May 25 11:17:34.973082 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:34.973068    2576 kubelet.go:953] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
May 25 11:17:34.973968 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:34.973957    2576 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
May 25 11:17:34.974005 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:34.973975    2576 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
May 25 11:17:34.974005 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:34.973982    2576 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
May 25 11:17:34.974005 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:34.973988    2576 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
May 25 11:17:34.974005 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:34.973998    2576 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
May 25 11:17:34.974123 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:34.974007    2576 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/secret"
May 25 11:17:34.974123 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:34.974015    2576 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
May 25 11:17:34.974123 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:34.974021    2576 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
May 25 11:17:34.974123 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:34.974028    2576 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/fc"
May 25 11:17:34.974123 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:34.974034    2576 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
May 25 11:17:34.974123 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:34.974043    2576 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/projected"
May 25 11:17:34.974481 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:34.974472    2576 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
May 25 11:17:34.975556 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:34.975545    2576 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/csi"
May 25 11:17:34.975595 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:34.975557    2576 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/image"
May 25 11:17:34.977052 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:34.977032    2576 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-9zqf8"
May 25 11:17:34.977507 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:17:34.977478    2576 reflector.go:200] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-132-254.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
May 25 11:17:34.977590 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:17:34.977484    2576 reflector.go:200] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
May 25 11:17:34.979047 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:34.979030    2576 csi_plugin.go:988] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-254.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
May 25 11:17:34.979154 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:34.979141    2576 watchdog_linux.go:99] "Systemd watchdog is not enabled"
May 25 11:17:34.979192 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:34.979180    2576 server.go:1295] "Started kubelet"
May 25 11:17:34.979313 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:34.979266    2576 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
May 25 11:17:34.979369 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:34.979254    2576 ratelimit.go:55] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
May 25 11:17:34.979419 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:34.979366    2576 server_v1.go:47] "podresources" method="list" useActivePods=true
May 25 11:17:34.979929 ip-10-0-132-254 systemd[1]: Started Kubernetes Kubelet.
May 25 11:17:34.981014 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:34.980998    2576 server.go:255] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
May 25 11:17:34.982170 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:34.982156    2576 server.go:317] "Adding debug handlers to kubelet server"
May 25 11:17:34.985630 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:34.985614    2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-9zqf8"
May 25 11:17:34.989887 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:17:34.989865    2576 kubelet.go:1618] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
May 25 11:17:34.990396 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:34.990377    2576 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
May 25 11:17:34.990471 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:34.990399    2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
May 25 11:17:34.990583 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:34.990568    2576 volume_manager.go:295] "The desired_state_of_world populator starts"
May 25 11:17:34.990639 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:34.990584    2576 volume_manager.go:297] "Starting Kubelet Volume Manager"
May 25 11:17:34.990719 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:34.990704    2576 desired_state_of_world_populator.go:150] "Desired state populator starts to run"
May 25 11:17:34.990847 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:34.990743    2576 reconstruct.go:97] "Volume reconstruction finished"
May 25 11:17:34.990847 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:34.990822    2576 reconciler.go:26] "Reconciler: start to sync state"
May 25 11:17:34.991320 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:34.991301    2576 factory.go:55] Registering systemd factory
May 25 11:17:34.991320 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:34.991322    2576 factory.go:223] Registration of the systemd container factory successfully
May 25 11:17:34.991440 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:17:34.991339    2576 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-132-254.ec2.internal\" not found"
May 25 11:17:34.991573 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:34.991559    2576 factory.go:153] Registering CRI-O factory
May 25 11:17:34.991646 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:34.991575    2576 factory.go:223] Registration of the crio container factory successfully
May 25 11:17:34.991646 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:34.991626    2576 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
May 25 11:17:34.991646 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:34.991645    2576 factory.go:103] Registering Raw factory
May 25 11:17:34.991789 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:34.991664    2576 manager.go:1196] Started watching for new ooms in manager
May 25 11:17:34.992548 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:34.992523    2576 manager.go:319] Starting recovery of all containers
May 25 11:17:34.993197 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:34.993181    2576 reflector.go:430] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
May 25 11:17:34.996071 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:17:34.996047    2576 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-132-254.ec2.internal\" not found" node="ip-10-0-132-254.ec2.internal"
May 25 11:17:35.003332 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:35.003312    2576 manager.go:324] Recovery completed
May 25 11:17:35.007559 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:35.007545    2576 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 25 11:17:35.011044 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:35.011026    2576 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-132-254.ec2.internal" event="NodeHasSufficientMemory"
May 25 11:17:35.011123 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:35.011061    2576 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-132-254.ec2.internal" event="NodeHasNoDiskPressure"
May 25 11:17:35.011123 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:35.011074    2576 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-132-254.ec2.internal" event="NodeHasSufficientPID"
May 25 11:17:35.011616 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:35.011601    2576 cpu_manager.go:222] "Starting CPU manager" policy="none"
May 25 11:17:35.011616 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:35.011614    2576 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
May 25 11:17:35.011704 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:35.011635    2576 state_mem.go:36] "Initialized new in-memory state store"
May 25 11:17:35.014661 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:35.014648    2576 policy_none.go:49] "None policy: Start"
May 25 11:17:35.014717 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:35.014668    2576 memory_manager.go:186] "Starting memorymanager" policy="None"
May 25 11:17:35.014717 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:35.014679    2576 state_mem.go:35] "Initializing new in-memory state store"
May 25 11:17:35.056987 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:35.056968    2576 manager.go:341] "Starting Device Plugin manager"
May 25 11:17:35.060365 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:17:35.057012    2576 manager.go:517] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
May 25 11:17:35.060365 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:35.057026    2576 server.go:85] "Starting device plugin registration server"
May 25 11:17:35.060365 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:35.057343    2576 eviction_manager.go:189] "Eviction manager: starting control loop"
May 25 11:17:35.060365 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:35.057358    2576 container_log_manager.go:189] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
May 25 11:17:35.060365 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:35.057461    2576 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
May 25 11:17:35.060365 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:35.057550    2576 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
May 25 11:17:35.060365 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:35.057559    2576 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
May 25 11:17:35.060365 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:17:35.058105    2576 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
May 25 11:17:35.060365 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:17:35.058142    2576 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-132-254.ec2.internal\" not found"
May 25 11:17:35.091773 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:35.091742    2576 kubelet_network_linux.go:49] "Initialized iptables rules." protocol="IPv4"
May 25 11:17:35.093106 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:35.093089    2576 kubelet_network_linux.go:49] "Initialized iptables rules." protocol="IPv6"
May 25 11:17:35.093174 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:35.093126    2576 status_manager.go:230] "Starting to sync pod status with apiserver"
May 25 11:17:35.093174 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:35.093144    2576 watchdog_linux.go:127] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
May 25 11:17:35.093174 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:35.093151    2576 kubelet.go:2451] "Starting kubelet main sync loop"
May 25 11:17:35.093350 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:17:35.093245    2576 kubelet.go:2475] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
May 25 11:17:35.095478 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:35.095460    2576 reflector.go:430] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
May 25 11:17:35.158317 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:35.158217    2576 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 25 11:17:35.159205 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:35.159189    2576 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-132-254.ec2.internal" event="NodeHasSufficientMemory"
May 25 11:17:35.159355 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:35.159223    2576 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-132-254.ec2.internal" event="NodeHasNoDiskPressure"
May 25 11:17:35.159355 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:35.159240    2576 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-132-254.ec2.internal" event="NodeHasSufficientPID"
May 25 11:17:35.159355 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:35.159272    2576 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-132-254.ec2.internal"
May 25 11:17:35.168118 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:35.168093    2576 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-132-254.ec2.internal"
May 25 11:17:35.168118 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:17:35.168118    2576 kubelet_node_status.go:597] "Error updating node status, will retry" err="error getting node \"ip-10-0-132-254.ec2.internal\": node \"ip-10-0-132-254.ec2.internal\" not found"
May 25 11:17:35.181708 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:17:35.181686    2576 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-132-254.ec2.internal\" not found"
May 25 11:17:35.194016 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:35.193981    2576 kubelet.go:2537] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-254.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-132-254.ec2.internal"]
May 25 11:17:35.194107 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:35.194087    2576 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 25 11:17:35.195502 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:35.195486    2576 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-132-254.ec2.internal" event="NodeHasSufficientMemory"
May 25 11:17:35.195579 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:35.195515    2576 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-132-254.ec2.internal" event="NodeHasNoDiskPressure"
May 25 11:17:35.195579 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:35.195530    2576 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-132-254.ec2.internal" event="NodeHasSufficientPID"
May 25 11:17:35.197226 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:35.197213    2576 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 25 11:17:35.197406 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:35.197391    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-254.ec2.internal"
May 25 11:17:35.197460 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:35.197420    2576 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 25 11:17:35.197966 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:35.197952    2576 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-132-254.ec2.internal" event="NodeHasSufficientMemory"
May 25 11:17:35.197966 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:35.197961    2576 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-132-254.ec2.internal" event="NodeHasSufficientMemory"
May 25 11:17:35.198059 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:35.197978    2576 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-132-254.ec2.internal" event="NodeHasNoDiskPressure"
May 25 11:17:35.198059 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:35.197990    2576 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-132-254.ec2.internal" event="NodeHasSufficientPID"
May 25 11:17:35.198059 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:35.197979    2576 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-132-254.ec2.internal" event="NodeHasNoDiskPressure"
May 25 11:17:35.198162 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:35.198065    2576 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-132-254.ec2.internal" event="NodeHasSufficientPID"
May 25 11:17:35.199663 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:35.199649    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-254.ec2.internal"
May 25 11:17:35.199731 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:35.199673    2576 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 25 11:17:35.200323 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:35.200307    2576 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-132-254.ec2.internal" event="NodeHasSufficientMemory"
May 25 11:17:35.200386 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:35.200337    2576 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-132-254.ec2.internal" event="NodeHasNoDiskPressure"
May 25 11:17:35.200386 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:35.200349    2576 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-132-254.ec2.internal" event="NodeHasSufficientPID"
May 25 11:17:35.227052 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:17:35.227029    2576 kubelet.go:3336] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-254.ec2.internal\" not found" node="ip-10-0-132-254.ec2.internal"
May 25 11:17:35.231478 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:17:35.231462    2576 kubelet.go:3336] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-254.ec2.internal\" not found" node="ip-10-0-132-254.ec2.internal"
May 25 11:17:35.281932 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:17:35.281903    2576 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-132-254.ec2.internal\" not found"
May 25 11:17:35.292476 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:35.292452    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/67ca951ca4e4e1d2d1949fbdcb1b1f35-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-254.ec2.internal\" (UID: \"67ca951ca4e4e1d2d1949fbdcb1b1f35\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-254.ec2.internal"
May 25 11:17:35.292607 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:35.292485    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/67ca951ca4e4e1d2d1949fbdcb1b1f35-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-254.ec2.internal\" (UID: \"67ca951ca4e4e1d2d1949fbdcb1b1f35\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-254.ec2.internal"
May 25 11:17:35.292607 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:35.292510    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/11df30495fdea7c5dad53acc2aaa0a94-config\") pod \"kube-apiserver-proxy-ip-10-0-132-254.ec2.internal\" (UID: \"11df30495fdea7c5dad53acc2aaa0a94\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-254.ec2.internal"
May 25 11:17:35.382841 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:17:35.382814    2576 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-132-254.ec2.internal\" not found"
May 25 11:17:35.393234 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:35.393210    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/67ca951ca4e4e1d2d1949fbdcb1b1f35-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-254.ec2.internal\" (UID: \"67ca951ca4e4e1d2d1949fbdcb1b1f35\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-254.ec2.internal"
May 25 11:17:35.393363 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:35.393179    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/67ca951ca4e4e1d2d1949fbdcb1b1f35-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-254.ec2.internal\" (UID: \"67ca951ca4e4e1d2d1949fbdcb1b1f35\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-254.ec2.internal"
May 25 11:17:35.393363 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:35.393331    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/67ca951ca4e4e1d2d1949fbdcb1b1f35-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-254.ec2.internal\" (UID: \"67ca951ca4e4e1d2d1949fbdcb1b1f35\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-254.ec2.internal"
May 25 11:17:35.393463 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:35.393361    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/11df30495fdea7c5dad53acc2aaa0a94-config\") pod \"kube-apiserver-proxy-ip-10-0-132-254.ec2.internal\" (UID: \"11df30495fdea7c5dad53acc2aaa0a94\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-254.ec2.internal"
May 25 11:17:35.393463 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:35.393369    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/67ca951ca4e4e1d2d1949fbdcb1b1f35-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-254.ec2.internal\" (UID: \"67ca951ca4e4e1d2d1949fbdcb1b1f35\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-254.ec2.internal"
May 25 11:17:35.393463 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:35.393405    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/11df30495fdea7c5dad53acc2aaa0a94-config\") pod \"kube-apiserver-proxy-ip-10-0-132-254.ec2.internal\" (UID: \"11df30495fdea7c5dad53acc2aaa0a94\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-254.ec2.internal"
May 25 11:17:35.483468 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:17:35.483398    2576 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-132-254.ec2.internal\" not found"
May 25 11:17:35.529883 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:35.529845    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-254.ec2.internal"
May 25 11:17:35.533710 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:35.533688    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-254.ec2.internal"
May 25 11:17:35.584259 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:17:35.584226    2576 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-132-254.ec2.internal\" not found"
May 25 11:17:35.684724 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:17:35.684703    2576 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-132-254.ec2.internal\" not found"
May 25 11:17:35.785270 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:17:35.785189    2576 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-132-254.ec2.internal\" not found"
May 25 11:17:35.883732 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:35.883696    2576 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
May 25 11:17:35.884428 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:35.883855    2576 reflector.go:556] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
May 25 11:17:35.884428 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:35.883888    2576 reflector.go:556] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
May 25 11:17:35.885858 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:17:35.885837    2576 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-132-254.ec2.internal\" not found"
May 25 11:17:35.909499 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:35.909463    2576 reflector.go:430] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
May 25 11:17:35.986344 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:17:35.986141    2576 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-132-254.ec2.internal\" not found"
May 25 11:17:35.988476 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:35.988448    2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-05-24 11:12:34 +0000 UTC" deadline="2028-03-01 12:23:52.791071828 +0000 UTC"
May 25 11:17:35.988476 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:35.988475    2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15505h6m16.802599222s"
May 25 11:17:35.990596 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:35.990580    2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
May 25 11:17:35.999437 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:35.999418    2576 reflector.go:430] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
May 25 11:17:36.023429 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:36.023347    2576 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-cdvg5"
May 25 11:17:36.029742 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:36.029720    2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-cdvg5"
May 25 11:17:36.087067 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:17:36.086965    2576 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-132-254.ec2.internal\" not found"
May 25 11:17:36.097192 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:36.097173    2576 reflector.go:430] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
May 25 11:17:36.125642 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:36.125614    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67ca951ca4e4e1d2d1949fbdcb1b1f35.slice/crio-3e421f4cdcc7a9c356538a485d5857a93908a5568dfc9203dfc6fab86055c212 WatchSource:0}: Error finding container 3e421f4cdcc7a9c356538a485d5857a93908a5568dfc9203dfc6fab86055c212: Status 404 returned error can't find the container with id 3e421f4cdcc7a9c356538a485d5857a93908a5568dfc9203dfc6fab86055c212
May 25 11:17:36.125875 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:36.125855    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11df30495fdea7c5dad53acc2aaa0a94.slice/crio-53906852dbc479714e669c770e39d0ce2868a89ed90ced9980fb6d1e7f2df771 WatchSource:0}: Error finding container 53906852dbc479714e669c770e39d0ce2868a89ed90ced9980fb6d1e7f2df771: Status 404 returned error can't find the container with id 53906852dbc479714e669c770e39d0ce2868a89ed90ced9980fb6d1e7f2df771
May 25 11:17:36.130121 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:36.130103    2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 25 11:17:36.191167 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:36.191143    2576 kubelet.go:3340] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-254.ec2.internal"
May 25 11:17:36.205077 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:36.205054    2576 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
May 25 11:17:36.206144 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:36.206132    2576 kubelet.go:3340] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-254.ec2.internal"
May 25 11:17:36.212975 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:36.212961    2576 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
May 25 11:17:36.963633 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:36.963590    2576 apiserver.go:52] "Watching apiserver"
May 25 11:17:36.970255 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:36.970228    2576 reflector.go:430] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
May 25 11:17:36.971490 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:36.971428    2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-qjgcp","openshift-multus/network-metrics-daemon-5859j","openshift-network-diagnostics/network-check-target-xmvkz","kube-system/kube-apiserver-proxy-ip-10-0-132-254.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-254.ec2.internal","openshift-multus/multus-additional-cni-plugins-dgspq","openshift-network-operator/iptables-alerter-qpwv2","openshift-ovn-kubernetes/ovnkube-node-h6b5t","kube-system/konnectivity-agent-4vc6k","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jrgpj","openshift-cluster-node-tuning-operator/tuned-tk5ng","openshift-dns/node-resolver-xn288","openshift-image-registry/node-ca-nkcf5"]
May 25 11:17:36.973981 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:36.973955    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jrgpj"
May 25 11:17:36.976371 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:36.976230    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-jl6ph\""
May 25 11:17:36.976484 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:36.976388    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
May 25 11:17:36.976542 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:36.976488    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
May 25 11:17:36.976600 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:36.976582    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
May 25 11:17:36.979254 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:36.979232    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5859j"
May 25 11:17:36.979371 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:17:36.979322    2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5859j" podUID="ff50c150-45a9-41c6-be52-886f2c960e85"
May 25 11:17:36.981494 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:36.981474    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xmvkz"
May 25 11:17:36.981589 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:17:36.981537    2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xmvkz" podUID="e77e329e-0279-4bf7-9210-7c8226f6a091"
May 25 11:17:36.983624 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:36.983601    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-dgspq"
May 25 11:17:36.983887 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:36.983786    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-qpwv2"
May 25 11:17:36.985943 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:36.985757    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
May 25 11:17:36.985943 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:36.985757    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
May 25 11:17:36.986089 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:36.986028    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-h6b5t"
May 25 11:17:36.986147 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:36.986117    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
May 25 11:17:36.986199 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:36.986168    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
May 25 11:17:36.986773 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:36.986289    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-b9qpb\""
May 25 11:17:36.986773 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:36.986450    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-pmns6\""
May 25 11:17:36.986773 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:36.986474    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
May 25 11:17:36.986773 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:36.986642    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
May 25 11:17:36.986773 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:36.986719    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
May 25 11:17:36.986773 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:36.986728    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
May 25 11:17:36.988254 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:36.988237    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-4vc6k"
May 25 11:17:36.988700 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:36.988682    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
May 25 11:17:36.988841 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:36.988823    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
May 25 11:17:36.988933 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:36.988878    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
May 25 11:17:36.988933 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:36.988919    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
May 25 11:17:36.989255 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:36.989235    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-fdwwv\""
May 25 11:17:36.989255 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:36.989246    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
May 25 11:17:36.990023 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:36.990006    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
May 25 11:17:36.990396 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:36.990379    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
May 25 11:17:36.990490 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:36.990449    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
May 25 11:17:36.990490 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:36.990478    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-kzk9c\""
May 25 11:17:36.994796 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:36.994693    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-qjgcp"
May 25 11:17:36.996641 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:36.996620    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
May 25 11:17:36.996734 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:36.996706    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-df4fj\""
May 25 11:17:36.997219 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:36.997185    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-tk5ng"
May 25 11:17:36.997322 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:36.997307    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-xn288"
May 25 11:17:36.999134 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:36.999113    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-l6vpk\""
May 25 11:17:36.999417 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:36.999357    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
May 25 11:17:36.999417 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:36.999376    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-xnrpf\""
May 25 11:17:36.999550 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:36.999453    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
May 25 11:17:36.999550 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:36.999472    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
May 25 11:17:36.999863 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:36.999846    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-nkcf5"
May 25 11:17:36.999953 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:36.999897    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
May 25 11:17:37.000373 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.000355    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a2602414-6828-4ead-beff-a1e1215f9851-cni-binary-copy\") pod \"multus-qjgcp\" (UID: \"a2602414-6828-4ead-beff-a1e1215f9851\") " pod="openshift-multus/multus-qjgcp"
May 25 11:17:37.000470 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.000384    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a2602414-6828-4ead-beff-a1e1215f9851-etc-kubernetes\") pod \"multus-qjgcp\" (UID: \"a2602414-6828-4ead-beff-a1e1215f9851\") " pod="openshift-multus/multus-qjgcp"
May 25 11:17:37.000470 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.000410    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gk8s2\" (UniqueName: \"kubernetes.io/projected/5ae22b47-9a06-49bb-a618-43df38833826-kube-api-access-gk8s2\") pod \"aws-ebs-csi-driver-node-jrgpj\" (UID: \"5ae22b47-9a06-49bb-a618-43df38833826\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jrgpj"
May 25 11:17:37.000470 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.000438    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqrzt\" (UniqueName: \"kubernetes.io/projected/ac23d02e-4fdd-4cfe-a16a-0958e2a7b62e-kube-api-access-cqrzt\") pod \"multus-additional-cni-plugins-dgspq\" (UID: \"ac23d02e-4fdd-4cfe-a16a-0958e2a7b62e\") " pod="openshift-multus/multus-additional-cni-plugins-dgspq"
May 25 11:17:37.000470 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.000463    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a2602414-6828-4ead-beff-a1e1215f9851-host-run-netns\") pod \"multus-qjgcp\" (UID: \"a2602414-6828-4ead-beff-a1e1215f9851\") " pod="openshift-multus/multus-qjgcp"
May 25 11:17:37.000650 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.000489    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a2602414-6828-4ead-beff-a1e1215f9851-multus-daemon-config\") pod \"multus-qjgcp\" (UID: \"a2602414-6828-4ead-beff-a1e1215f9851\") " pod="openshift-multus/multus-qjgcp"
May 25 11:17:37.000650 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.000514    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/20a93003-13be-468a-aeb8-bfd0bcb86dbd-etc-systemd\") pod \"tuned-tk5ng\" (UID: \"20a93003-13be-468a-aeb8-bfd0bcb86dbd\") " pod="openshift-cluster-node-tuning-operator/tuned-tk5ng"
May 25 11:17:37.000650 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.000537    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/20a93003-13be-468a-aeb8-bfd0bcb86dbd-tmp\") pod \"tuned-tk5ng\" (UID: \"20a93003-13be-468a-aeb8-bfd0bcb86dbd\") " pod="openshift-cluster-node-tuning-operator/tuned-tk5ng"
May 25 11:17:37.000650 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.000581    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/12b45353-8444-41fc-a8ed-fb691764eb63-env-overrides\") pod \"ovnkube-node-h6b5t\" (UID: \"12b45353-8444-41fc-a8ed-fb691764eb63\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6b5t"
May 25 11:17:37.000650 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.000614    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a2602414-6828-4ead-beff-a1e1215f9851-multus-socket-dir-parent\") pod \"multus-qjgcp\" (UID: \"a2602414-6828-4ead-beff-a1e1215f9851\") " pod="openshift-multus/multus-qjgcp"
May 25 11:17:37.000650 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.000639    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5ae22b47-9a06-49bb-a618-43df38833826-device-dir\") pod \"aws-ebs-csi-driver-node-jrgpj\" (UID: \"5ae22b47-9a06-49bb-a618-43df38833826\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jrgpj"
May 25 11:17:37.000929 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.000667    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ff50c150-45a9-41c6-be52-886f2c960e85-metrics-certs\") pod \"network-metrics-daemon-5859j\" (UID: \"ff50c150-45a9-41c6-be52-886f2c960e85\") " pod="openshift-multus/network-metrics-daemon-5859j"
May 25 11:17:37.000929 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.000691    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5fnt\" (UniqueName: \"kubernetes.io/projected/e77e329e-0279-4bf7-9210-7c8226f6a091-kube-api-access-w5fnt\") pod \"network-check-target-xmvkz\" (UID: \"e77e329e-0279-4bf7-9210-7c8226f6a091\") " pod="openshift-network-diagnostics/network-check-target-xmvkz"
May 25 11:17:37.000929 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.000715    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ac23d02e-4fdd-4cfe-a16a-0958e2a7b62e-os-release\") pod \"multus-additional-cni-plugins-dgspq\" (UID: \"ac23d02e-4fdd-4cfe-a16a-0958e2a7b62e\") " pod="openshift-multus/multus-additional-cni-plugins-dgspq"
May 25 11:17:37.000929 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.000738    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/12b45353-8444-41fc-a8ed-fb691764eb63-run-openvswitch\") pod \"ovnkube-node-h6b5t\" (UID: \"12b45353-8444-41fc-a8ed-fb691764eb63\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6b5t"
May 25 11:17:37.000929 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.000761    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/12b45353-8444-41fc-a8ed-fb691764eb63-host-cni-netd\") pod \"ovnkube-node-h6b5t\" (UID: \"12b45353-8444-41fc-a8ed-fb691764eb63\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6b5t"
May 25 11:17:37.000929 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.000787    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ac23d02e-4fdd-4cfe-a16a-0958e2a7b62e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-dgspq\" (UID: \"ac23d02e-4fdd-4cfe-a16a-0958e2a7b62e\") " pod="openshift-multus/multus-additional-cni-plugins-dgspq"
May 25 11:17:37.000929 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.000812    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/12b45353-8444-41fc-a8ed-fb691764eb63-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-h6b5t\" (UID: \"12b45353-8444-41fc-a8ed-fb691764eb63\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6b5t"
May 25 11:17:37.000929 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.000836    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/20a93003-13be-468a-aeb8-bfd0bcb86dbd-var-lib-kubelet\") pod \"tuned-tk5ng\" (UID: \"20a93003-13be-468a-aeb8-bfd0bcb86dbd\") " pod="openshift-cluster-node-tuning-operator/tuned-tk5ng"
May 25 11:17:37.000929 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.000859    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5ae22b47-9a06-49bb-a618-43df38833826-etc-selinux\") pod \"aws-ebs-csi-driver-node-jrgpj\" (UID: \"5ae22b47-9a06-49bb-a618-43df38833826\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jrgpj"
May 25 11:17:37.000929 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.000882    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a2602414-6828-4ead-beff-a1e1215f9851-multus-cni-dir\") pod \"multus-qjgcp\" (UID: \"a2602414-6828-4ead-beff-a1e1215f9851\") " pod="openshift-multus/multus-qjgcp"
May 25 11:17:37.001383 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.000930    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a2602414-6828-4ead-beff-a1e1215f9851-os-release\") pod \"multus-qjgcp\" (UID: \"a2602414-6828-4ead-beff-a1e1215f9851\") " pod="openshift-multus/multus-qjgcp"
May 25 11:17:37.001383 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.000967    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a2602414-6828-4ead-beff-a1e1215f9851-host-var-lib-cni-multus\") pod \"multus-qjgcp\" (UID: \"a2602414-6828-4ead-beff-a1e1215f9851\") " pod="openshift-multus/multus-qjgcp"
May 25 11:17:37.001383 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.000998    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/20a93003-13be-468a-aeb8-bfd0bcb86dbd-etc-sysctl-d\") pod \"tuned-tk5ng\" (UID: \"20a93003-13be-468a-aeb8-bfd0bcb86dbd\") " pod="openshift-cluster-node-tuning-operator/tuned-tk5ng"
May 25 11:17:37.001383 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.001023    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/20a93003-13be-468a-aeb8-bfd0bcb86dbd-sys\") pod \"tuned-tk5ng\" (UID: \"20a93003-13be-468a-aeb8-bfd0bcb86dbd\") " pod="openshift-cluster-node-tuning-operator/tuned-tk5ng"
May 25 11:17:37.001383 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.001047    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ac23d02e-4fdd-4cfe-a16a-0958e2a7b62e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-dgspq\" (UID: \"ac23d02e-4fdd-4cfe-a16a-0958e2a7b62e\") " pod="openshift-multus/multus-additional-cni-plugins-dgspq"
May 25 11:17:37.001383 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.001074    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/12b45353-8444-41fc-a8ed-fb691764eb63-ovn-node-metrics-cert\") pod \"ovnkube-node-h6b5t\" (UID: \"12b45353-8444-41fc-a8ed-fb691764eb63\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6b5t"
May 25 11:17:37.001383 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.001107    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a2602414-6828-4ead-beff-a1e1215f9851-host-run-k8s-cni-cncf-io\") pod \"multus-qjgcp\" (UID: \"a2602414-6828-4ead-beff-a1e1215f9851\") " pod="openshift-multus/multus-qjgcp"
May 25 11:17:37.001383 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.001135    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/20a93003-13be-468a-aeb8-bfd0bcb86dbd-host\") pod \"tuned-tk5ng\" (UID: \"20a93003-13be-468a-aeb8-bfd0bcb86dbd\") " pod="openshift-cluster-node-tuning-operator/tuned-tk5ng"
May 25 11:17:37.001383 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.001171    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/20a93003-13be-468a-aeb8-bfd0bcb86dbd-etc-tuned\") pod \"tuned-tk5ng\" (UID: \"20a93003-13be-468a-aeb8-bfd0bcb86dbd\") " pod="openshift-cluster-node-tuning-operator/tuned-tk5ng"
May 25 11:17:37.001383 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.001209    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/12b45353-8444-41fc-a8ed-fb691764eb63-ovnkube-script-lib\") pod \"ovnkube-node-h6b5t\" (UID: \"12b45353-8444-41fc-a8ed-fb691764eb63\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6b5t"
May 25 11:17:37.001383 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.001249    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5ae22b47-9a06-49bb-a618-43df38833826-socket-dir\") pod \"aws-ebs-csi-driver-node-jrgpj\" (UID: \"5ae22b47-9a06-49bb-a618-43df38833826\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jrgpj"
May 25 11:17:37.001383 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.001308    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a2602414-6828-4ead-beff-a1e1215f9851-hostroot\") pod \"multus-qjgcp\" (UID: \"a2602414-6828-4ead-beff-a1e1215f9851\") " pod="openshift-multus/multus-qjgcp"
May 25 11:17:37.001383 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.001341    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txvjb\" (UniqueName: \"kubernetes.io/projected/20a93003-13be-468a-aeb8-bfd0bcb86dbd-kube-api-access-txvjb\") pod \"tuned-tk5ng\" (UID: \"20a93003-13be-468a-aeb8-bfd0bcb86dbd\") " pod="openshift-cluster-node-tuning-operator/tuned-tk5ng"
May 25 11:17:37.001383 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.001365    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5ae22b47-9a06-49bb-a618-43df38833826-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jrgpj\" (UID: \"5ae22b47-9a06-49bb-a618-43df38833826\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jrgpj"
May 25 11:17:37.001979 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.001391    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/12b45353-8444-41fc-a8ed-fb691764eb63-node-log\") pod \"ovnkube-node-h6b5t\" (UID: \"12b45353-8444-41fc-a8ed-fb691764eb63\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6b5t"
May 25 11:17:37.001979 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.001414    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/deccfe91-b8ff-47cc-92b2-d6f19d19ebec-agent-certs\") pod \"konnectivity-agent-4vc6k\" (UID: \"deccfe91-b8ff-47cc-92b2-d6f19d19ebec\") " pod="kube-system/konnectivity-agent-4vc6k"
May 25 11:17:37.001979 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.001438    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/20a93003-13be-468a-aeb8-bfd0bcb86dbd-etc-sysctl-conf\") pod \"tuned-tk5ng\" (UID: \"20a93003-13be-468a-aeb8-bfd0bcb86dbd\") " pod="openshift-cluster-node-tuning-operator/tuned-tk5ng"
May 25 11:17:37.001979 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.001490    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/20a93003-13be-468a-aeb8-bfd0bcb86dbd-lib-modules\") pod \"tuned-tk5ng\" (UID: \"20a93003-13be-468a-aeb8-bfd0bcb86dbd\") " pod="openshift-cluster-node-tuning-operator/tuned-tk5ng"
May 25 11:17:37.001979 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.001524    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5ae22b47-9a06-49bb-a618-43df38833826-registration-dir\") pod \"aws-ebs-csi-driver-node-jrgpj\" (UID: \"5ae22b47-9a06-49bb-a618-43df38833826\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jrgpj"
May 25 11:17:37.001979 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.001558    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ac23d02e-4fdd-4cfe-a16a-0958e2a7b62e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-dgspq\" (UID: \"ac23d02e-4fdd-4cfe-a16a-0958e2a7b62e\") " pod="openshift-multus/multus-additional-cni-plugins-dgspq"
May 25 11:17:37.001979 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.001588    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/6d2e36b9-21f2-4f1b-91ee-4c6348cded20-iptables-alerter-script\") pod \"iptables-alerter-qpwv2\" (UID: \"6d2e36b9-21f2-4f1b-91ee-4c6348cded20\") " pod="openshift-network-operator/iptables-alerter-qpwv2"
May 25 11:17:37.001979 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.001601    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
May 25 11:17:37.001979 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.001614    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/12b45353-8444-41fc-a8ed-fb691764eb63-systemd-units\") pod \"ovnkube-node-h6b5t\" (UID: \"12b45353-8444-41fc-a8ed-fb691764eb63\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6b5t"
May 25 11:17:37.001979 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.001636    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/12b45353-8444-41fc-a8ed-fb691764eb63-host-run-netns\") pod \"ovnkube-node-h6b5t\" (UID: \"12b45353-8444-41fc-a8ed-fb691764eb63\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6b5t"
May 25 11:17:37.001979 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.001675    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a2602414-6828-4ead-beff-a1e1215f9851-host-var-lib-kubelet\") pod \"multus-qjgcp\" (UID: \"a2602414-6828-4ead-beff-a1e1215f9851\") " pod="openshift-multus/multus-qjgcp"
May 25 11:17:37.001979 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.001706    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56vfx\" (UniqueName: \"kubernetes.io/projected/a2602414-6828-4ead-beff-a1e1215f9851-kube-api-access-56vfx\") pod \"multus-qjgcp\" (UID: \"a2602414-6828-4ead-beff-a1e1215f9851\") " pod="openshift-multus/multus-qjgcp"
May 25 11:17:37.001979 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.001737    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6d2e36b9-21f2-4f1b-91ee-4c6348cded20-host-slash\") pod \"iptables-alerter-qpwv2\" (UID: \"6d2e36b9-21f2-4f1b-91ee-4c6348cded20\") " pod="openshift-network-operator/iptables-alerter-qpwv2"
May 25 11:17:37.001979 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.001766    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/12b45353-8444-41fc-a8ed-fb691764eb63-host-run-ovn-kubernetes\") pod \"ovnkube-node-h6b5t\" (UID: \"12b45353-8444-41fc-a8ed-fb691764eb63\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6b5t"
May 25 11:17:37.001979 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.001816    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5ae22b47-9a06-49bb-a618-43df38833826-sys-fs\") pod \"aws-ebs-csi-driver-node-jrgpj\" (UID: \"5ae22b47-9a06-49bb-a618-43df38833826\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jrgpj"
May 25 11:17:37.001979 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.001834    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lp9kp\" (UniqueName: \"kubernetes.io/projected/ff50c150-45a9-41c6-be52-886f2c960e85-kube-api-access-lp9kp\") pod \"network-metrics-daemon-5859j\" (UID: \"ff50c150-45a9-41c6-be52-886f2c960e85\") " pod="openshift-multus/network-metrics-daemon-5859j"
May 25 11:17:37.001979 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.001838    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
May 25 11:17:37.002718 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.001854    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ac23d02e-4fdd-4cfe-a16a-0958e2a7b62e-cni-binary-copy\") pod \"multus-additional-cni-plugins-dgspq\" (UID: \"ac23d02e-4fdd-4cfe-a16a-0958e2a7b62e\") " pod="openshift-multus/multus-additional-cni-plugins-dgspq"
May 25 11:17:37.002718 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.001879    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vsrnp\" (UniqueName: \"kubernetes.io/projected/6d2e36b9-21f2-4f1b-91ee-4c6348cded20-kube-api-access-vsrnp\") pod \"iptables-alerter-qpwv2\" (UID: \"6d2e36b9-21f2-4f1b-91ee-4c6348cded20\") " pod="openshift-network-operator/iptables-alerter-qpwv2"
May 25 11:17:37.002718 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.001904    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/12b45353-8444-41fc-a8ed-fb691764eb63-host-cni-bin\") pod \"ovnkube-node-h6b5t\" (UID: \"12b45353-8444-41fc-a8ed-fb691764eb63\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6b5t"
May 25 11:17:37.002718 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.001931    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a2602414-6828-4ead-beff-a1e1215f9851-system-cni-dir\") pod \"multus-qjgcp\" (UID: \"a2602414-6828-4ead-beff-a1e1215f9851\") " pod="openshift-multus/multus-qjgcp"
May 25 11:17:37.002718 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.001972    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/20a93003-13be-468a-aeb8-bfd0bcb86dbd-etc-modprobe-d\") pod \"tuned-tk5ng\" (UID: \"20a93003-13be-468a-aeb8-bfd0bcb86dbd\") " pod="openshift-cluster-node-tuning-operator/tuned-tk5ng"
May 25 11:17:37.002718 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.001997    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/12b45353-8444-41fc-a8ed-fb691764eb63-var-lib-openvswitch\") pod \"ovnkube-node-h6b5t\" (UID: \"12b45353-8444-41fc-a8ed-fb691764eb63\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6b5t"
May 25 11:17:37.002718 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.002022    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/12b45353-8444-41fc-a8ed-fb691764eb63-log-socket\") pod \"ovnkube-node-h6b5t\" (UID: \"12b45353-8444-41fc-a8ed-fb691764eb63\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6b5t"
May 25 11:17:37.002718 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.002054    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/20a93003-13be-468a-aeb8-bfd0bcb86dbd-etc-kubernetes\") pod \"tuned-tk5ng\" (UID: \"20a93003-13be-468a-aeb8-bfd0bcb86dbd\") " pod="openshift-cluster-node-tuning-operator/tuned-tk5ng"
May 25 11:17:37.002718 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.002075    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/12b45353-8444-41fc-a8ed-fb691764eb63-host-slash\") pod \"ovnkube-node-h6b5t\" (UID: \"12b45353-8444-41fc-a8ed-fb691764eb63\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6b5t"
May 25 11:17:37.002718 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.002092    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-2hp4h\""
May 25 11:17:37.002718 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.002098    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/12b45353-8444-41fc-a8ed-fb691764eb63-run-systemd\") pod \"ovnkube-node-h6b5t\" (UID: \"12b45353-8444-41fc-a8ed-fb691764eb63\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6b5t"
May 25 11:17:37.002718 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.002124    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqd7v\" (UniqueName: \"kubernetes.io/projected/12b45353-8444-41fc-a8ed-fb691764eb63-kube-api-access-rqd7v\") pod \"ovnkube-node-h6b5t\" (UID: \"12b45353-8444-41fc-a8ed-fb691764eb63\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6b5t"
May 25 11:17:37.002718 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.002145    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a2602414-6828-4ead-beff-a1e1215f9851-multus-conf-dir\") pod \"multus-qjgcp\" (UID: \"a2602414-6828-4ead-beff-a1e1215f9851\") " pod="openshift-multus/multus-qjgcp"
May 25 11:17:37.002718 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.002147    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
May 25 11:17:37.002718 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.002168    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a2602414-6828-4ead-beff-a1e1215f9851-host-run-multus-certs\") pod \"multus-qjgcp\" (UID: \"a2602414-6828-4ead-beff-a1e1215f9851\") " pod="openshift-multus/multus-qjgcp"
May 25 11:17:37.002718 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.002192    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/20a93003-13be-468a-aeb8-bfd0bcb86dbd-etc-sysconfig\") pod \"tuned-tk5ng\" (UID: \"20a93003-13be-468a-aeb8-bfd0bcb86dbd\") " pod="openshift-cluster-node-tuning-operator/tuned-tk5ng"
May 25 11:17:37.002718 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.002215    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ac23d02e-4fdd-4cfe-a16a-0958e2a7b62e-system-cni-dir\") pod \"multus-additional-cni-plugins-dgspq\" (UID: \"ac23d02e-4fdd-4cfe-a16a-0958e2a7b62e\") " pod="openshift-multus/multus-additional-cni-plugins-dgspq"
May 25 11:17:37.003341 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.002267    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/12b45353-8444-41fc-a8ed-fb691764eb63-etc-openvswitch\") pod \"ovnkube-node-h6b5t\" (UID: \"12b45353-8444-41fc-a8ed-fb691764eb63\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6b5t"
May 25 11:17:37.003341 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.002323    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/12b45353-8444-41fc-a8ed-fb691764eb63-run-ovn\") pod \"ovnkube-node-h6b5t\" (UID: \"12b45353-8444-41fc-a8ed-fb691764eb63\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6b5t"
May 25 11:17:37.003341 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.002378    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a2602414-6828-4ead-beff-a1e1215f9851-host-var-lib-cni-bin\") pod \"multus-qjgcp\" (UID: \"a2602414-6828-4ead-beff-a1e1215f9851\") " pod="openshift-multus/multus-qjgcp"
May 25 11:17:37.003341 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.002404    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ac23d02e-4fdd-4cfe-a16a-0958e2a7b62e-cnibin\") pod \"multus-additional-cni-plugins-dgspq\" (UID: \"ac23d02e-4fdd-4cfe-a16a-0958e2a7b62e\") " pod="openshift-multus/multus-additional-cni-plugins-dgspq"
May 25 11:17:37.003341 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.002420    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/12b45353-8444-41fc-a8ed-fb691764eb63-host-kubelet\") pod \"ovnkube-node-h6b5t\" (UID: \"12b45353-8444-41fc-a8ed-fb691764eb63\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6b5t"
May 25 11:17:37.003341 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.002434    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a2602414-6828-4ead-beff-a1e1215f9851-cnibin\") pod \"multus-qjgcp\" (UID: \"a2602414-6828-4ead-beff-a1e1215f9851\") " pod="openshift-multus/multus-qjgcp"
May 25 11:17:37.003341 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.002451    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/20a93003-13be-468a-aeb8-bfd0bcb86dbd-run\") pod \"tuned-tk5ng\" (UID: \"20a93003-13be-468a-aeb8-bfd0bcb86dbd\") " pod="openshift-cluster-node-tuning-operator/tuned-tk5ng"
May 25 11:17:37.003341 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.002473    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/12b45353-8444-41fc-a8ed-fb691764eb63-ovnkube-config\") pod \"ovnkube-node-h6b5t\" (UID: \"12b45353-8444-41fc-a8ed-fb691764eb63\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6b5t"
May 25 11:17:37.003341 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.002496    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/deccfe91-b8ff-47cc-92b2-d6f19d19ebec-konnectivity-ca\") pod \"konnectivity-agent-4vc6k\" (UID: \"deccfe91-b8ff-47cc-92b2-d6f19d19ebec\") " pod="kube-system/konnectivity-agent-4vc6k"
May 25 11:17:37.031245 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.031212    2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-05-24 11:12:36 +0000 UTC" deadline="2028-02-15 03:26:28.734831104 +0000 UTC"
May 25 11:17:37.031245 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.031241    2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15136h8m51.703592867s"
May 25 11:17:37.092495 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.092457    2576 desired_state_of_world_populator.go:158] "Finished populating initial desired state of world"
May 25 11:17:37.097092 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.097038    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-254.ec2.internal" event={"ID":"67ca951ca4e4e1d2d1949fbdcb1b1f35","Type":"ContainerStarted","Data":"3e421f4cdcc7a9c356538a485d5857a93908a5568dfc9203dfc6fab86055c212"}
May 25 11:17:37.098091 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.098067    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-254.ec2.internal" event={"ID":"11df30495fdea7c5dad53acc2aaa0a94","Type":"ContainerStarted","Data":"53906852dbc479714e669c770e39d0ce2868a89ed90ced9980fb6d1e7f2df771"}
May 25 11:17:37.103128 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.103099    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/12b45353-8444-41fc-a8ed-fb691764eb63-host-kubelet\") pod \"ovnkube-node-h6b5t\" (UID: \"12b45353-8444-41fc-a8ed-fb691764eb63\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6b5t"
May 25 11:17:37.103128 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.103128    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a2602414-6828-4ead-beff-a1e1215f9851-cnibin\") pod \"multus-qjgcp\" (UID: \"a2602414-6828-4ead-beff-a1e1215f9851\") " pod="openshift-multus/multus-qjgcp"
May 25 11:17:37.103271 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.103144    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/20a93003-13be-468a-aeb8-bfd0bcb86dbd-run\") pod \"tuned-tk5ng\" (UID: \"20a93003-13be-468a-aeb8-bfd0bcb86dbd\") " pod="openshift-cluster-node-tuning-operator/tuned-tk5ng"
May 25 11:17:37.103271 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.103159    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/12b45353-8444-41fc-a8ed-fb691764eb63-ovnkube-config\") pod \"ovnkube-node-h6b5t\" (UID: \"12b45353-8444-41fc-a8ed-fb691764eb63\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6b5t"
May 25 11:17:37.103271 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.103174    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/deccfe91-b8ff-47cc-92b2-d6f19d19ebec-konnectivity-ca\") pod \"konnectivity-agent-4vc6k\" (UID: \"deccfe91-b8ff-47cc-92b2-d6f19d19ebec\") " pod="kube-system/konnectivity-agent-4vc6k"
May 25 11:17:37.103271 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.103209    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a2602414-6828-4ead-beff-a1e1215f9851-cni-binary-copy\") pod \"multus-qjgcp\" (UID: \"a2602414-6828-4ead-beff-a1e1215f9851\") " pod="openshift-multus/multus-qjgcp"
May 25 11:17:37.103271 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.103235    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a2602414-6828-4ead-beff-a1e1215f9851-cnibin\") pod \"multus-qjgcp\" (UID: \"a2602414-6828-4ead-beff-a1e1215f9851\") " pod="openshift-multus/multus-qjgcp"
May 25 11:17:37.103271 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.103211    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/12b45353-8444-41fc-a8ed-fb691764eb63-host-kubelet\") pod \"ovnkube-node-h6b5t\" (UID: \"12b45353-8444-41fc-a8ed-fb691764eb63\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6b5t"
May 25 11:17:37.103271 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.103252    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a2602414-6828-4ead-beff-a1e1215f9851-etc-kubernetes\") pod \"multus-qjgcp\" (UID: \"a2602414-6828-4ead-beff-a1e1215f9851\") " pod="openshift-multus/multus-qjgcp"
May 25 11:17:37.103612 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.103295    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cd7634d4-8eb9-42a5-9e01-7d94645afbd3-host\") pod \"node-ca-nkcf5\" (UID: \"cd7634d4-8eb9-42a5-9e01-7d94645afbd3\") " pod="openshift-image-registry/node-ca-nkcf5"
May 25 11:17:37.103612 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.103316    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a2602414-6828-4ead-beff-a1e1215f9851-etc-kubernetes\") pod \"multus-qjgcp\" (UID: \"a2602414-6828-4ead-beff-a1e1215f9851\") " pod="openshift-multus/multus-qjgcp"
May 25 11:17:37.103612 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.103321    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55bzk\" (UniqueName: \"kubernetes.io/projected/cd7634d4-8eb9-42a5-9e01-7d94645afbd3-kube-api-access-55bzk\") pod \"node-ca-nkcf5\" (UID: \"cd7634d4-8eb9-42a5-9e01-7d94645afbd3\") " pod="openshift-image-registry/node-ca-nkcf5"
May 25 11:17:37.103612 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.103327    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/20a93003-13be-468a-aeb8-bfd0bcb86dbd-run\") pod \"tuned-tk5ng\" (UID: \"20a93003-13be-468a-aeb8-bfd0bcb86dbd\") " pod="openshift-cluster-node-tuning-operator/tuned-tk5ng"
May 25 11:17:37.103612 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.103347    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djb6n\" (UniqueName: \"kubernetes.io/projected/4ab64dbc-4ab2-4199-bf57-698ec5539507-kube-api-access-djb6n\") pod \"node-resolver-xn288\" (UID: \"4ab64dbc-4ab2-4199-bf57-698ec5539507\") " pod="openshift-dns/node-resolver-xn288"
May 25 11:17:37.103612 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.103373    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-gk8s2\" (UniqueName: \"kubernetes.io/projected/5ae22b47-9a06-49bb-a618-43df38833826-kube-api-access-gk8s2\") pod \"aws-ebs-csi-driver-node-jrgpj\" (UID: \"5ae22b47-9a06-49bb-a618-43df38833826\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jrgpj"
May 25 11:17:37.103612 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.103398    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-cqrzt\" (UniqueName: \"kubernetes.io/projected/ac23d02e-4fdd-4cfe-a16a-0958e2a7b62e-kube-api-access-cqrzt\") pod \"multus-additional-cni-plugins-dgspq\" (UID: \"ac23d02e-4fdd-4cfe-a16a-0958e2a7b62e\") " pod="openshift-multus/multus-additional-cni-plugins-dgspq"
May 25 11:17:37.103612 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.103421    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a2602414-6828-4ead-beff-a1e1215f9851-host-run-netns\") pod \"multus-qjgcp\" (UID: \"a2602414-6828-4ead-beff-a1e1215f9851\") " pod="openshift-multus/multus-qjgcp"
May 25 11:17:37.103612 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.103504    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a2602414-6828-4ead-beff-a1e1215f9851-host-run-netns\") pod \"multus-qjgcp\" (UID: \"a2602414-6828-4ead-beff-a1e1215f9851\") " pod="openshift-multus/multus-qjgcp"
May 25 11:17:37.104026 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.103617    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a2602414-6828-4ead-beff-a1e1215f9851-multus-daemon-config\") pod \"multus-qjgcp\" (UID: \"a2602414-6828-4ead-beff-a1e1215f9851\") " pod="openshift-multus/multus-qjgcp"
May 25 11:17:37.104026 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.103654    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/20a93003-13be-468a-aeb8-bfd0bcb86dbd-etc-systemd\") pod \"tuned-tk5ng\" (UID: \"20a93003-13be-468a-aeb8-bfd0bcb86dbd\") " pod="openshift-cluster-node-tuning-operator/tuned-tk5ng"
May 25 11:17:37.104026 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.103680    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/20a93003-13be-468a-aeb8-bfd0bcb86dbd-tmp\") pod \"tuned-tk5ng\" (UID: \"20a93003-13be-468a-aeb8-bfd0bcb86dbd\") " pod="openshift-cluster-node-tuning-operator/tuned-tk5ng"
May 25 11:17:37.104026 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.103709    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ab64dbc-4ab2-4199-bf57-698ec5539507-tmp-dir\") pod \"node-resolver-xn288\" (UID: \"4ab64dbc-4ab2-4199-bf57-698ec5539507\") " pod="openshift-dns/node-resolver-xn288"
May 25 11:17:37.104026 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.103735    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/12b45353-8444-41fc-a8ed-fb691764eb63-env-overrides\") pod \"ovnkube-node-h6b5t\" (UID: \"12b45353-8444-41fc-a8ed-fb691764eb63\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6b5t"
May 25 11:17:37.104026 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.103751    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/20a93003-13be-468a-aeb8-bfd0bcb86dbd-etc-systemd\") pod \"tuned-tk5ng\" (UID: \"20a93003-13be-468a-aeb8-bfd0bcb86dbd\") " pod="openshift-cluster-node-tuning-operator/tuned-tk5ng"
May 25 11:17:37.104026 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.103758    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a2602414-6828-4ead-beff-a1e1215f9851-multus-socket-dir-parent\") pod \"multus-qjgcp\" (UID: \"a2602414-6828-4ead-beff-a1e1215f9851\") " pod="openshift-multus/multus-qjgcp"
May 25 11:17:37.104026 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.103795    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5ae22b47-9a06-49bb-a618-43df38833826-device-dir\") pod \"aws-ebs-csi-driver-node-jrgpj\" (UID: \"5ae22b47-9a06-49bb-a618-43df38833826\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jrgpj"
May 25 11:17:37.104026 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.103811    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a2602414-6828-4ead-beff-a1e1215f9851-multus-socket-dir-parent\") pod \"multus-qjgcp\" (UID: \"a2602414-6828-4ead-beff-a1e1215f9851\") " pod="openshift-multus/multus-qjgcp"
May 25 11:17:37.104026 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.103824    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ff50c150-45a9-41c6-be52-886f2c960e85-metrics-certs\") pod \"network-metrics-daemon-5859j\" (UID: \"ff50c150-45a9-41c6-be52-886f2c960e85\") " pod="openshift-multus/network-metrics-daemon-5859j"
May 25 11:17:37.104026 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.103851    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-w5fnt\" (UniqueName: \"kubernetes.io/projected/e77e329e-0279-4bf7-9210-7c8226f6a091-kube-api-access-w5fnt\") pod \"network-check-target-xmvkz\" (UID: \"e77e329e-0279-4bf7-9210-7c8226f6a091\") " pod="openshift-network-diagnostics/network-check-target-xmvkz"
May 25 11:17:37.104026 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.103880    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ac23d02e-4fdd-4cfe-a16a-0958e2a7b62e-os-release\") pod \"multus-additional-cni-plugins-dgspq\" (UID: \"ac23d02e-4fdd-4cfe-a16a-0958e2a7b62e\") " pod="openshift-multus/multus-additional-cni-plugins-dgspq"
May 25 11:17:37.104026 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.103885    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/deccfe91-b8ff-47cc-92b2-d6f19d19ebec-konnectivity-ca\") pod \"konnectivity-agent-4vc6k\" (UID: \"deccfe91-b8ff-47cc-92b2-d6f19d19ebec\") " pod="kube-system/konnectivity-agent-4vc6k"
May 25 11:17:37.104026 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.103894    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a2602414-6828-4ead-beff-a1e1215f9851-cni-binary-copy\") pod \"multus-qjgcp\" (UID: \"a2602414-6828-4ead-beff-a1e1215f9851\") " pod="openshift-multus/multus-qjgcp"
May 25 11:17:37.104026 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.103911    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/12b45353-8444-41fc-a8ed-fb691764eb63-run-openvswitch\") pod \"ovnkube-node-h6b5t\" (UID: \"12b45353-8444-41fc-a8ed-fb691764eb63\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6b5t"
May 25 11:17:37.104026 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.103915    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/12b45353-8444-41fc-a8ed-fb691764eb63-ovnkube-config\") pod \"ovnkube-node-h6b5t\" (UID: \"12b45353-8444-41fc-a8ed-fb691764eb63\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6b5t"
May 25 11:17:37.104026 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.103951    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/12b45353-8444-41fc-a8ed-fb691764eb63-host-cni-netd\") pod \"ovnkube-node-h6b5t\" (UID: \"12b45353-8444-41fc-a8ed-fb691764eb63\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6b5t"
May 25 11:17:37.104811 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.103952    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/12b45353-8444-41fc-a8ed-fb691764eb63-run-openvswitch\") pod \"ovnkube-node-h6b5t\" (UID: \"12b45353-8444-41fc-a8ed-fb691764eb63\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6b5t"
May 25 11:17:37.104811 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.103988    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ac23d02e-4fdd-4cfe-a16a-0958e2a7b62e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-dgspq\" (UID: \"ac23d02e-4fdd-4cfe-a16a-0958e2a7b62e\") " pod="openshift-multus/multus-additional-cni-plugins-dgspq"
May 25 11:17:37.104811 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.104007    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/12b45353-8444-41fc-a8ed-fb691764eb63-host-cni-netd\") pod \"ovnkube-node-h6b5t\" (UID: \"12b45353-8444-41fc-a8ed-fb691764eb63\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6b5t"
May 25 11:17:37.104811 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.104014    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/12b45353-8444-41fc-a8ed-fb691764eb63-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-h6b5t\" (UID: \"12b45353-8444-41fc-a8ed-fb691764eb63\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6b5t"
May 25 11:17:37.104811 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:17:37.104023    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 11:17:37.104811 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.104041    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/20a93003-13be-468a-aeb8-bfd0bcb86dbd-var-lib-kubelet\") pod \"tuned-tk5ng\" (UID: \"20a93003-13be-468a-aeb8-bfd0bcb86dbd\") " pod="openshift-cluster-node-tuning-operator/tuned-tk5ng"
May 25 11:17:37.104811 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.104069    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5ae22b47-9a06-49bb-a618-43df38833826-etc-selinux\") pod \"aws-ebs-csi-driver-node-jrgpj\" (UID: \"5ae22b47-9a06-49bb-a618-43df38833826\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jrgpj"
May 25 11:17:37.104811 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.104071    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ac23d02e-4fdd-4cfe-a16a-0958e2a7b62e-os-release\") pod \"multus-additional-cni-plugins-dgspq\" (UID: \"ac23d02e-4fdd-4cfe-a16a-0958e2a7b62e\") " pod="openshift-multus/multus-additional-cni-plugins-dgspq"
May 25 11:17:37.104811 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:17:37.104104    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ff50c150-45a9-41c6-be52-886f2c960e85-metrics-certs podName:ff50c150-45a9-41c6-be52-886f2c960e85 nodeName:}" failed. No retries permitted until 2026-05-25 11:17:37.604074603 +0000 UTC m=+3.121806054 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ff50c150-45a9-41c6-be52-886f2c960e85-metrics-certs") pod "network-metrics-daemon-5859j" (UID: "ff50c150-45a9-41c6-be52-886f2c960e85") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 11:17:37.104811 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.104122    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5ae22b47-9a06-49bb-a618-43df38833826-etc-selinux\") pod \"aws-ebs-csi-driver-node-jrgpj\" (UID: \"5ae22b47-9a06-49bb-a618-43df38833826\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jrgpj"
May 25 11:17:37.104811 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.104146    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5ae22b47-9a06-49bb-a618-43df38833826-device-dir\") pod \"aws-ebs-csi-driver-node-jrgpj\" (UID: \"5ae22b47-9a06-49bb-a618-43df38833826\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jrgpj"
May 25 11:17:37.104811 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.104151    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a2602414-6828-4ead-beff-a1e1215f9851-multus-daemon-config\") pod \"multus-qjgcp\" (UID: \"a2602414-6828-4ead-beff-a1e1215f9851\") " pod="openshift-multus/multus-qjgcp"
May 25 11:17:37.104811 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.104192    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ac23d02e-4fdd-4cfe-a16a-0958e2a7b62e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-dgspq\" (UID: \"ac23d02e-4fdd-4cfe-a16a-0958e2a7b62e\") " pod="openshift-multus/multus-additional-cni-plugins-dgspq"
May 25 11:17:37.104811 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.104215    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/20a93003-13be-468a-aeb8-bfd0bcb86dbd-var-lib-kubelet\") pod \"tuned-tk5ng\" (UID: \"20a93003-13be-468a-aeb8-bfd0bcb86dbd\") " pod="openshift-cluster-node-tuning-operator/tuned-tk5ng"
May 25 11:17:37.104811 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.104188    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a2602414-6828-4ead-beff-a1e1215f9851-multus-cni-dir\") pod \"multus-qjgcp\" (UID: \"a2602414-6828-4ead-beff-a1e1215f9851\") " pod="openshift-multus/multus-qjgcp"
May 25 11:17:37.104811 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.104216    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/12b45353-8444-41fc-a8ed-fb691764eb63-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-h6b5t\" (UID: \"12b45353-8444-41fc-a8ed-fb691764eb63\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6b5t"
May 25 11:17:37.104811 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.104246    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a2602414-6828-4ead-beff-a1e1215f9851-os-release\") pod \"multus-qjgcp\" (UID: \"a2602414-6828-4ead-beff-a1e1215f9851\") " pod="openshift-multus/multus-qjgcp"
May 25 11:17:37.105564 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.104247    2576 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
May 25 11:17:37.105564 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.104264    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a2602414-6828-4ead-beff-a1e1215f9851-multus-cni-dir\") pod \"multus-qjgcp\" (UID: \"a2602414-6828-4ead-beff-a1e1215f9851\") " pod="openshift-multus/multus-qjgcp"
May 25 11:17:37.105564 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.104272    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a2602414-6828-4ead-beff-a1e1215f9851-host-var-lib-cni-multus\") pod \"multus-qjgcp\" (UID: \"a2602414-6828-4ead-beff-a1e1215f9851\") " pod="openshift-multus/multus-qjgcp"
May 25 11:17:37.105564 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.104324    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/20a93003-13be-468a-aeb8-bfd0bcb86dbd-etc-sysctl-d\") pod \"tuned-tk5ng\" (UID: \"20a93003-13be-468a-aeb8-bfd0bcb86dbd\") " pod="openshift-cluster-node-tuning-operator/tuned-tk5ng"
May 25 11:17:37.105564 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.104342    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a2602414-6828-4ead-beff-a1e1215f9851-os-release\") pod \"multus-qjgcp\" (UID: \"a2602414-6828-4ead-beff-a1e1215f9851\") " pod="openshift-multus/multus-qjgcp"
May 25 11:17:37.105564 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.104347    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a2602414-6828-4ead-beff-a1e1215f9851-host-var-lib-cni-multus\") pod \"multus-qjgcp\" (UID: \"a2602414-6828-4ead-beff-a1e1215f9851\") " pod="openshift-multus/multus-qjgcp"
May 25 11:17:37.105564 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.104348    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/20a93003-13be-468a-aeb8-bfd0bcb86dbd-sys\") pod \"tuned-tk5ng\" (UID: \"20a93003-13be-468a-aeb8-bfd0bcb86dbd\") " pod="openshift-cluster-node-tuning-operator/tuned-tk5ng"
May 25 11:17:37.105564 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.104380    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ac23d02e-4fdd-4cfe-a16a-0958e2a7b62e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-dgspq\" (UID: \"ac23d02e-4fdd-4cfe-a16a-0958e2a7b62e\") " pod="openshift-multus/multus-additional-cni-plugins-dgspq"
May 25 11:17:37.105564 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.104385    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/20a93003-13be-468a-aeb8-bfd0bcb86dbd-sys\") pod \"tuned-tk5ng\" (UID: \"20a93003-13be-468a-aeb8-bfd0bcb86dbd\") " pod="openshift-cluster-node-tuning-operator/tuned-tk5ng"
May 25 11:17:37.105564 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.104461    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/20a93003-13be-468a-aeb8-bfd0bcb86dbd-etc-sysctl-d\") pod \"tuned-tk5ng\" (UID: \"20a93003-13be-468a-aeb8-bfd0bcb86dbd\") " pod="openshift-cluster-node-tuning-operator/tuned-tk5ng"
May 25 11:17:37.105564 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.104494    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/12b45353-8444-41fc-a8ed-fb691764eb63-ovn-node-metrics-cert\") pod \"ovnkube-node-h6b5t\" (UID: \"12b45353-8444-41fc-a8ed-fb691764eb63\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6b5t"
May 25 11:17:37.105564 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.104526    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a2602414-6828-4ead-beff-a1e1215f9851-host-run-k8s-cni-cncf-io\") pod \"multus-qjgcp\" (UID: \"a2602414-6828-4ead-beff-a1e1215f9851\") " pod="openshift-multus/multus-qjgcp"
May 25 11:17:37.105564 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.104553    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/20a93003-13be-468a-aeb8-bfd0bcb86dbd-host\") pod \"tuned-tk5ng\" (UID: \"20a93003-13be-468a-aeb8-bfd0bcb86dbd\") " pod="openshift-cluster-node-tuning-operator/tuned-tk5ng"
May 25 11:17:37.105564 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.104582    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/20a93003-13be-468a-aeb8-bfd0bcb86dbd-etc-tuned\") pod \"tuned-tk5ng\" (UID: \"20a93003-13be-468a-aeb8-bfd0bcb86dbd\") " pod="openshift-cluster-node-tuning-operator/tuned-tk5ng"
May 25 11:17:37.105564 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.104596    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a2602414-6828-4ead-beff-a1e1215f9851-host-run-k8s-cni-cncf-io\") pod \"multus-qjgcp\" (UID: \"a2602414-6828-4ead-beff-a1e1215f9851\") " pod="openshift-multus/multus-qjgcp"
May 25 11:17:37.105564 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.104608    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/12b45353-8444-41fc-a8ed-fb691764eb63-ovnkube-script-lib\") pod \"ovnkube-node-h6b5t\" (UID: \"12b45353-8444-41fc-a8ed-fb691764eb63\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6b5t"
May 25 11:17:37.105564 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.104663    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/20a93003-13be-468a-aeb8-bfd0bcb86dbd-host\") pod \"tuned-tk5ng\" (UID: \"20a93003-13be-468a-aeb8-bfd0bcb86dbd\") " pod="openshift-cluster-node-tuning-operator/tuned-tk5ng"
May 25 11:17:37.105564 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.104603    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/12b45353-8444-41fc-a8ed-fb691764eb63-env-overrides\") pod \"ovnkube-node-h6b5t\" (UID: \"12b45353-8444-41fc-a8ed-fb691764eb63\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6b5t"
May 25 11:17:37.106397 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.104713    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5ae22b47-9a06-49bb-a618-43df38833826-socket-dir\") pod \"aws-ebs-csi-driver-node-jrgpj\" (UID: \"5ae22b47-9a06-49bb-a618-43df38833826\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jrgpj"
May 25 11:17:37.106397 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.104741    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a2602414-6828-4ead-beff-a1e1215f9851-hostroot\") pod \"multus-qjgcp\" (UID: \"a2602414-6828-4ead-beff-a1e1215f9851\") " pod="openshift-multus/multus-qjgcp"
May 25 11:17:37.106397 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.104766    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-txvjb\" (UniqueName: \"kubernetes.io/projected/20a93003-13be-468a-aeb8-bfd0bcb86dbd-kube-api-access-txvjb\") pod \"tuned-tk5ng\" (UID: \"20a93003-13be-468a-aeb8-bfd0bcb86dbd\") " pod="openshift-cluster-node-tuning-operator/tuned-tk5ng"
May 25 11:17:37.106397 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.104794    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5ae22b47-9a06-49bb-a618-43df38833826-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jrgpj\" (UID: \"5ae22b47-9a06-49bb-a618-43df38833826\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jrgpj"
May 25 11:17:37.106397 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.104818    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/12b45353-8444-41fc-a8ed-fb691764eb63-node-log\") pod \"ovnkube-node-h6b5t\" (UID: \"12b45353-8444-41fc-a8ed-fb691764eb63\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6b5t"
May 25 11:17:37.106397 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.104841    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/deccfe91-b8ff-47cc-92b2-d6f19d19ebec-agent-certs\") pod \"konnectivity-agent-4vc6k\" (UID: \"deccfe91-b8ff-47cc-92b2-d6f19d19ebec\") " pod="kube-system/konnectivity-agent-4vc6k"
May 25 11:17:37.106397 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.104867    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/20a93003-13be-468a-aeb8-bfd0bcb86dbd-etc-sysctl-conf\") pod \"tuned-tk5ng\" (UID: \"20a93003-13be-468a-aeb8-bfd0bcb86dbd\") " pod="openshift-cluster-node-tuning-operator/tuned-tk5ng"
May 25 11:17:37.106397 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.104891    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/20a93003-13be-468a-aeb8-bfd0bcb86dbd-lib-modules\") pod \"tuned-tk5ng\" (UID: \"20a93003-13be-468a-aeb8-bfd0bcb86dbd\") " pod="openshift-cluster-node-tuning-operator/tuned-tk5ng"
May 25 11:17:37.106397 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.104904    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ac23d02e-4fdd-4cfe-a16a-0958e2a7b62e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-dgspq\" (UID: \"ac23d02e-4fdd-4cfe-a16a-0958e2a7b62e\") " pod="openshift-multus/multus-additional-cni-plugins-dgspq"
May 25 11:17:37.106397 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.104918    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5ae22b47-9a06-49bb-a618-43df38833826-registration-dir\") pod \"aws-ebs-csi-driver-node-jrgpj\" (UID: \"5ae22b47-9a06-49bb-a618-43df38833826\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jrgpj"
May 25 11:17:37.106397 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.104943    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ac23d02e-4fdd-4cfe-a16a-0958e2a7b62e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-dgspq\" (UID: \"ac23d02e-4fdd-4cfe-a16a-0958e2a7b62e\") " pod="openshift-multus/multus-additional-cni-plugins-dgspq"
May 25 11:17:37.106397 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.104970    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5ae22b47-9a06-49bb-a618-43df38833826-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jrgpj\" (UID: \"5ae22b47-9a06-49bb-a618-43df38833826\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jrgpj"
May 25 11:17:37.106397 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.104973    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/6d2e36b9-21f2-4f1b-91ee-4c6348cded20-iptables-alerter-script\") pod \"iptables-alerter-qpwv2\" (UID: \"6d2e36b9-21f2-4f1b-91ee-4c6348cded20\") " pod="openshift-network-operator/iptables-alerter-qpwv2"
May 25 11:17:37.106397 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.105003    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/12b45353-8444-41fc-a8ed-fb691764eb63-systemd-units\") pod \"ovnkube-node-h6b5t\" (UID: \"12b45353-8444-41fc-a8ed-fb691764eb63\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6b5t"
May 25 11:17:37.106397 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.105028    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/12b45353-8444-41fc-a8ed-fb691764eb63-host-run-netns\") pod \"ovnkube-node-h6b5t\" (UID: \"12b45353-8444-41fc-a8ed-fb691764eb63\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6b5t"
May 25 11:17:37.106397 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.105048    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/12b45353-8444-41fc-a8ed-fb691764eb63-ovnkube-script-lib\") pod \"ovnkube-node-h6b5t\" (UID: \"12b45353-8444-41fc-a8ed-fb691764eb63\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6b5t"
May 25 11:17:37.106397 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.105053    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a2602414-6828-4ead-beff-a1e1215f9851-host-var-lib-kubelet\") pod \"multus-qjgcp\" (UID: \"a2602414-6828-4ead-beff-a1e1215f9851\") " pod="openshift-multus/multus-qjgcp"
May 25 11:17:37.107159 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.105079    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-56vfx\" (UniqueName: \"kubernetes.io/projected/a2602414-6828-4ead-beff-a1e1215f9851-kube-api-access-56vfx\") pod \"multus-qjgcp\" (UID: \"a2602414-6828-4ead-beff-a1e1215f9851\") " pod="openshift-multus/multus-qjgcp"
May 25 11:17:37.107159 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.105104    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6d2e36b9-21f2-4f1b-91ee-4c6348cded20-host-slash\") pod \"iptables-alerter-qpwv2\" (UID: \"6d2e36b9-21f2-4f1b-91ee-4c6348cded20\") " pod="openshift-network-operator/iptables-alerter-qpwv2"
May 25 11:17:37.107159 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.105132    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/12b45353-8444-41fc-a8ed-fb691764eb63-host-run-ovn-kubernetes\") pod \"ovnkube-node-h6b5t\" (UID: \"12b45353-8444-41fc-a8ed-fb691764eb63\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6b5t"
May 25 11:17:37.107159 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.105159    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5ae22b47-9a06-49bb-a618-43df38833826-sys-fs\") pod \"aws-ebs-csi-driver-node-jrgpj\" (UID: \"5ae22b47-9a06-49bb-a618-43df38833826\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jrgpj"
May 25 11:17:37.107159 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.105186    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-lp9kp\" (UniqueName: \"kubernetes.io/projected/ff50c150-45a9-41c6-be52-886f2c960e85-kube-api-access-lp9kp\") pod \"network-metrics-daemon-5859j\" (UID: \"ff50c150-45a9-41c6-be52-886f2c960e85\") " pod="openshift-multus/network-metrics-daemon-5859j"
May 25 11:17:37.107159 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.105213    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ac23d02e-4fdd-4cfe-a16a-0958e2a7b62e-cni-binary-copy\") pod \"multus-additional-cni-plugins-dgspq\" (UID: \"ac23d02e-4fdd-4cfe-a16a-0958e2a7b62e\") " pod="openshift-multus/multus-additional-cni-plugins-dgspq"
May 25 11:17:37.107159 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.105240    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-vsrnp\" (UniqueName: \"kubernetes.io/projected/6d2e36b9-21f2-4f1b-91ee-4c6348cded20-kube-api-access-vsrnp\") pod \"iptables-alerter-qpwv2\" (UID: \"6d2e36b9-21f2-4f1b-91ee-4c6348cded20\") " pod="openshift-network-operator/iptables-alerter-qpwv2"
May 25 11:17:37.107159 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.105264    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/12b45353-8444-41fc-a8ed-fb691764eb63-host-cni-bin\") pod \"ovnkube-node-h6b5t\" (UID: \"12b45353-8444-41fc-a8ed-fb691764eb63\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6b5t"
May 25 11:17:37.107159 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.105353    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a2602414-6828-4ead-beff-a1e1215f9851-hostroot\") pod \"multus-qjgcp\" (UID: \"a2602414-6828-4ead-beff-a1e1215f9851\") " pod="openshift-multus/multus-qjgcp"
May 25 11:17:37.107159 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.105355    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a2602414-6828-4ead-beff-a1e1215f9851-system-cni-dir\") pod \"multus-qjgcp\" (UID: \"a2602414-6828-4ead-beff-a1e1215f9851\") " pod="openshift-multus/multus-qjgcp"
May 25 11:17:37.107159 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.105390    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/20a93003-13be-468a-aeb8-bfd0bcb86dbd-etc-modprobe-d\") pod \"tuned-tk5ng\" (UID: \"20a93003-13be-468a-aeb8-bfd0bcb86dbd\") " pod="openshift-cluster-node-tuning-operator/tuned-tk5ng"
May 25 11:17:37.107159 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.105404    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a2602414-6828-4ead-beff-a1e1215f9851-system-cni-dir\") pod \"multus-qjgcp\" (UID: \"a2602414-6828-4ead-beff-a1e1215f9851\") " pod="openshift-multus/multus-qjgcp"
May 25 11:17:37.107159 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.105408    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/cd7634d4-8eb9-42a5-9e01-7d94645afbd3-serviceca\") pod \"node-ca-nkcf5\" (UID: \"cd7634d4-8eb9-42a5-9e01-7d94645afbd3\") " pod="openshift-image-registry/node-ca-nkcf5"
May 25 11:17:37.107159 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.105447    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/12b45353-8444-41fc-a8ed-fb691764eb63-var-lib-openvswitch\") pod \"ovnkube-node-h6b5t\" (UID: \"12b45353-8444-41fc-a8ed-fb691764eb63\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6b5t"
May 25 11:17:37.107159 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.105453    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/6d2e36b9-21f2-4f1b-91ee-4c6348cded20-iptables-alerter-script\") pod \"iptables-alerter-qpwv2\" (UID: \"6d2e36b9-21f2-4f1b-91ee-4c6348cded20\") " pod="openshift-network-operator/iptables-alerter-qpwv2"
May 25 11:17:37.107159 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.105469    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/20a93003-13be-468a-aeb8-bfd0bcb86dbd-etc-modprobe-d\") pod \"tuned-tk5ng\" (UID: \"20a93003-13be-468a-aeb8-bfd0bcb86dbd\") " pod="openshift-cluster-node-tuning-operator/tuned-tk5ng"
May 25 11:17:37.107159 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.105474    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/12b45353-8444-41fc-a8ed-fb691764eb63-log-socket\") pod \"ovnkube-node-h6b5t\" (UID: \"12b45353-8444-41fc-a8ed-fb691764eb63\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6b5t"
May 25 11:17:37.108002 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.105499    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/20a93003-13be-468a-aeb8-bfd0bcb86dbd-etc-kubernetes\") pod \"tuned-tk5ng\" (UID: \"20a93003-13be-468a-aeb8-bfd0bcb86dbd\") " pod="openshift-cluster-node-tuning-operator/tuned-tk5ng"
May 25 11:17:37.108002 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.105506    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/12b45353-8444-41fc-a8ed-fb691764eb63-node-log\") pod \"ovnkube-node-h6b5t\" (UID: \"12b45353-8444-41fc-a8ed-fb691764eb63\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6b5t"
May 25 11:17:37.108002 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.105530    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/4ab64dbc-4ab2-4199-bf57-698ec5539507-hosts-file\") pod \"node-resolver-xn288\" (UID: \"4ab64dbc-4ab2-4199-bf57-698ec5539507\") " pod="openshift-dns/node-resolver-xn288"
May 25 11:17:37.108002 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.105538    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/20a93003-13be-468a-aeb8-bfd0bcb86dbd-lib-modules\") pod \"tuned-tk5ng\" (UID: \"20a93003-13be-468a-aeb8-bfd0bcb86dbd\") " pod="openshift-cluster-node-tuning-operator/tuned-tk5ng"
May 25 11:17:37.108002 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.105556    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5ae22b47-9a06-49bb-a618-43df38833826-registration-dir\") pod \"aws-ebs-csi-driver-node-jrgpj\" (UID: \"5ae22b47-9a06-49bb-a618-43df38833826\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jrgpj"
May 25 11:17:37.108002 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.105560    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/12b45353-8444-41fc-a8ed-fb691764eb63-host-slash\") pod \"ovnkube-node-h6b5t\" (UID: \"12b45353-8444-41fc-a8ed-fb691764eb63\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6b5t"
May 25 11:17:37.108002 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.105586    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/12b45353-8444-41fc-a8ed-fb691764eb63-run-systemd\") pod \"ovnkube-node-h6b5t\" (UID: \"12b45353-8444-41fc-a8ed-fb691764eb63\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6b5t"
May 25 11:17:37.108002 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.105615    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-rqd7v\" (UniqueName: \"kubernetes.io/projected/12b45353-8444-41fc-a8ed-fb691764eb63-kube-api-access-rqd7v\") pod \"ovnkube-node-h6b5t\" (UID: \"12b45353-8444-41fc-a8ed-fb691764eb63\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6b5t"
May 25 11:17:37.108002 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.105641    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a2602414-6828-4ead-beff-a1e1215f9851-multus-conf-dir\") pod \"multus-qjgcp\" (UID: \"a2602414-6828-4ead-beff-a1e1215f9851\") " pod="openshift-multus/multus-qjgcp"
May 25 11:17:37.108002 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.105667    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a2602414-6828-4ead-beff-a1e1215f9851-host-run-multus-certs\") pod \"multus-qjgcp\" (UID: \"a2602414-6828-4ead-beff-a1e1215f9851\") " pod="openshift-multus/multus-qjgcp"
May 25 11:17:37.108002 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.105693    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/20a93003-13be-468a-aeb8-bfd0bcb86dbd-etc-sysconfig\") pod \"tuned-tk5ng\" (UID: \"20a93003-13be-468a-aeb8-bfd0bcb86dbd\") " pod="openshift-cluster-node-tuning-operator/tuned-tk5ng"
May 25 11:17:37.108002 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.105721    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ac23d02e-4fdd-4cfe-a16a-0958e2a7b62e-system-cni-dir\") pod \"multus-additional-cni-plugins-dgspq\" (UID: \"ac23d02e-4fdd-4cfe-a16a-0958e2a7b62e\") " pod="openshift-multus/multus-additional-cni-plugins-dgspq"
May 25 11:17:37.108002 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.105747    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/12b45353-8444-41fc-a8ed-fb691764eb63-etc-openvswitch\") pod \"ovnkube-node-h6b5t\" (UID: \"12b45353-8444-41fc-a8ed-fb691764eb63\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6b5t"
May 25 11:17:37.108002 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.105780    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/12b45353-8444-41fc-a8ed-fb691764eb63-run-ovn\") pod \"ovnkube-node-h6b5t\" (UID: \"12b45353-8444-41fc-a8ed-fb691764eb63\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6b5t"
May 25 11:17:37.108002 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.105807    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a2602414-6828-4ead-beff-a1e1215f9851-host-var-lib-cni-bin\") pod \"multus-qjgcp\" (UID: \"a2602414-6828-4ead-beff-a1e1215f9851\") " pod="openshift-multus/multus-qjgcp"
May 25 11:17:37.108002 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.105828    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/12b45353-8444-41fc-a8ed-fb691764eb63-var-lib-openvswitch\") pod \"ovnkube-node-h6b5t\" (UID: \"12b45353-8444-41fc-a8ed-fb691764eb63\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6b5t"
May 25 11:17:37.108002 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.105834    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ac23d02e-4fdd-4cfe-a16a-0958e2a7b62e-cnibin\") pod \"multus-additional-cni-plugins-dgspq\" (UID: \"ac23d02e-4fdd-4cfe-a16a-0958e2a7b62e\") " pod="openshift-multus/multus-additional-cni-plugins-dgspq"
May 25 11:17:37.108763 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.105862    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/12b45353-8444-41fc-a8ed-fb691764eb63-systemd-units\") pod \"ovnkube-node-h6b5t\" (UID: \"12b45353-8444-41fc-a8ed-fb691764eb63\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6b5t"
May 25 11:17:37.108763 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.105886    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/12b45353-8444-41fc-a8ed-fb691764eb63-host-run-netns\") pod \"ovnkube-node-h6b5t\" (UID: \"12b45353-8444-41fc-a8ed-fb691764eb63\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6b5t"
May 25 11:17:37.108763 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.105907    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a2602414-6828-4ead-beff-a1e1215f9851-host-var-lib-kubelet\") pod \"multus-qjgcp\" (UID: \"a2602414-6828-4ead-beff-a1e1215f9851\") " pod="openshift-multus/multus-qjgcp"
May 25 11:17:37.108763 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.105914    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ac23d02e-4fdd-4cfe-a16a-0958e2a7b62e-cnibin\") pod \"multus-additional-cni-plugins-dgspq\" (UID: \"ac23d02e-4fdd-4cfe-a16a-0958e2a7b62e\") " pod="openshift-multus/multus-additional-cni-plugins-dgspq"
May 25 11:17:37.108763 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.105940    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6d2e36b9-21f2-4f1b-91ee-4c6348cded20-host-slash\") pod \"iptables-alerter-qpwv2\" (UID: \"6d2e36b9-21f2-4f1b-91ee-4c6348cded20\") " pod="openshift-network-operator/iptables-alerter-qpwv2"
May 25 11:17:37.108763 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.105960    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ac23d02e-4fdd-4cfe-a16a-0958e2a7b62e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-dgspq\" (UID: \"ac23d02e-4fdd-4cfe-a16a-0958e2a7b62e\") " pod="openshift-multus/multus-additional-cni-plugins-dgspq"
May 25 11:17:37.108763 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.105984    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/12b45353-8444-41fc-a8ed-fb691764eb63-host-run-ovn-kubernetes\") pod \"ovnkube-node-h6b5t\" (UID: \"12b45353-8444-41fc-a8ed-fb691764eb63\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6b5t"
May 25 11:17:37.108763 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.106018    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5ae22b47-9a06-49bb-a618-43df38833826-sys-fs\") pod \"aws-ebs-csi-driver-node-jrgpj\" (UID: \"5ae22b47-9a06-49bb-a618-43df38833826\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jrgpj"
May 25 11:17:37.108763 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.106130    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a2602414-6828-4ead-beff-a1e1215f9851-multus-conf-dir\") pod \"multus-qjgcp\" (UID: \"a2602414-6828-4ead-beff-a1e1215f9851\") " pod="openshift-multus/multus-qjgcp"
May 25 11:17:37.108763 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.106197    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/20a93003-13be-468a-aeb8-bfd0bcb86dbd-etc-sysconfig\") pod \"tuned-tk5ng\" (UID: \"20a93003-13be-468a-aeb8-bfd0bcb86dbd\") " pod="openshift-cluster-node-tuning-operator/tuned-tk5ng"
May 25 11:17:37.108763 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.106243    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/12b45353-8444-41fc-a8ed-fb691764eb63-log-socket\") pod \"ovnkube-node-h6b5t\" (UID: \"12b45353-8444-41fc-a8ed-fb691764eb63\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6b5t"
May 25 11:17:37.108763 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.106312    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/20a93003-13be-468a-aeb8-bfd0bcb86dbd-etc-kubernetes\") pod \"tuned-tk5ng\" (UID: \"20a93003-13be-468a-aeb8-bfd0bcb86dbd\") " pod="openshift-cluster-node-tuning-operator/tuned-tk5ng"
May 25 11:17:37.108763 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.106359    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/12b45353-8444-41fc-a8ed-fb691764eb63-host-cni-bin\") pod \"ovnkube-node-h6b5t\" (UID: \"12b45353-8444-41fc-a8ed-fb691764eb63\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6b5t"
May 25 11:17:37.108763 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.106368    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/12b45353-8444-41fc-a8ed-fb691764eb63-host-slash\") pod \"ovnkube-node-h6b5t\" (UID: \"12b45353-8444-41fc-a8ed-fb691764eb63\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6b5t"
May 25 11:17:37.108763 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.106408    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/12b45353-8444-41fc-a8ed-fb691764eb63-etc-openvswitch\") pod \"ovnkube-node-h6b5t\" (UID: \"12b45353-8444-41fc-a8ed-fb691764eb63\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6b5t"
May 25 11:17:37.108763 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.106399    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ac23d02e-4fdd-4cfe-a16a-0958e2a7b62e-system-cni-dir\") pod \"multus-additional-cni-plugins-dgspq\" (UID: \"ac23d02e-4fdd-4cfe-a16a-0958e2a7b62e\") " pod="openshift-multus/multus-additional-cni-plugins-dgspq"
May 25 11:17:37.108763 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.106445    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/12b45353-8444-41fc-a8ed-fb691764eb63-run-ovn\") pod \"ovnkube-node-h6b5t\" (UID: \"12b45353-8444-41fc-a8ed-fb691764eb63\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6b5t"
May 25 11:17:37.108763 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.106462    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/12b45353-8444-41fc-a8ed-fb691764eb63-run-systemd\") pod \"ovnkube-node-h6b5t\" (UID: \"12b45353-8444-41fc-a8ed-fb691764eb63\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6b5t"
May 25 11:17:37.109382 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.105321    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5ae22b47-9a06-49bb-a618-43df38833826-socket-dir\") pod \"aws-ebs-csi-driver-node-jrgpj\" (UID: \"5ae22b47-9a06-49bb-a618-43df38833826\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jrgpj"
May 25 11:17:37.109382 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.106489    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a2602414-6828-4ead-beff-a1e1215f9851-host-var-lib-cni-bin\") pod \"multus-qjgcp\" (UID: \"a2602414-6828-4ead-beff-a1e1215f9851\") " pod="openshift-multus/multus-qjgcp"
May 25 11:17:37.109382 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.106530    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a2602414-6828-4ead-beff-a1e1215f9851-host-run-multus-certs\") pod \"multus-qjgcp\" (UID: \"a2602414-6828-4ead-beff-a1e1215f9851\") " pod="openshift-multus/multus-qjgcp"
May 25 11:17:37.109382 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.106575    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/20a93003-13be-468a-aeb8-bfd0bcb86dbd-etc-sysctl-conf\") pod \"tuned-tk5ng\" (UID: \"20a93003-13be-468a-aeb8-bfd0bcb86dbd\") " pod="openshift-cluster-node-tuning-operator/tuned-tk5ng"
May 25 11:17:37.109382 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.106610    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ac23d02e-4fdd-4cfe-a16a-0958e2a7b62e-cni-binary-copy\") pod \"multus-additional-cni-plugins-dgspq\" (UID: \"ac23d02e-4fdd-4cfe-a16a-0958e2a7b62e\") " pod="openshift-multus/multus-additional-cni-plugins-dgspq"
May 25 11:17:37.109382 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.108247    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/12b45353-8444-41fc-a8ed-fb691764eb63-ovn-node-metrics-cert\") pod \"ovnkube-node-h6b5t\" (UID: \"12b45353-8444-41fc-a8ed-fb691764eb63\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6b5t"
May 25 11:17:37.109382 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.108371    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/20a93003-13be-468a-aeb8-bfd0bcb86dbd-etc-tuned\") pod \"tuned-tk5ng\" (UID: \"20a93003-13be-468a-aeb8-bfd0bcb86dbd\") " pod="openshift-cluster-node-tuning-operator/tuned-tk5ng"
May 25 11:17:37.109382 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.108680    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/deccfe91-b8ff-47cc-92b2-d6f19d19ebec-agent-certs\") pod \"konnectivity-agent-4vc6k\" (UID: \"deccfe91-b8ff-47cc-92b2-d6f19d19ebec\") " pod="kube-system/konnectivity-agent-4vc6k"
May 25 11:17:37.109382 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.109038    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/20a93003-13be-468a-aeb8-bfd0bcb86dbd-tmp\") pod \"tuned-tk5ng\" (UID: \"20a93003-13be-468a-aeb8-bfd0bcb86dbd\") " pod="openshift-cluster-node-tuning-operator/tuned-tk5ng"
May 25 11:17:37.114713 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:17:37.114605    2576 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 25 11:17:37.114713 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:17:37.114625    2576 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 25 11:17:37.114713 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:17:37.114638    2576 projected.go:194] Error preparing data for projected volume kube-api-access-w5fnt for pod openshift-network-diagnostics/network-check-target-xmvkz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 11:17:37.114713 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:17:37.114712    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e77e329e-0279-4bf7-9210-7c8226f6a091-kube-api-access-w5fnt podName:e77e329e-0279-4bf7-9210-7c8226f6a091 nodeName:}" failed. No retries permitted until 2026-05-25 11:17:37.614692448 +0000 UTC m=+3.132423908 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-w5fnt" (UniqueName: "kubernetes.io/projected/e77e329e-0279-4bf7-9210-7c8226f6a091-kube-api-access-w5fnt") pod "network-check-target-xmvkz" (UID: "e77e329e-0279-4bf7-9210-7c8226f6a091") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 11:17:37.117118 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.117093    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqrzt\" (UniqueName: \"kubernetes.io/projected/ac23d02e-4fdd-4cfe-a16a-0958e2a7b62e-kube-api-access-cqrzt\") pod \"multus-additional-cni-plugins-dgspq\" (UID: \"ac23d02e-4fdd-4cfe-a16a-0958e2a7b62e\") " pod="openshift-multus/multus-additional-cni-plugins-dgspq"
May 25 11:17:37.117481 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.117456    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gk8s2\" (UniqueName: \"kubernetes.io/projected/5ae22b47-9a06-49bb-a618-43df38833826-kube-api-access-gk8s2\") pod \"aws-ebs-csi-driver-node-jrgpj\" (UID: \"5ae22b47-9a06-49bb-a618-43df38833826\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jrgpj"
May 25 11:17:37.117571 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.117551    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-txvjb\" (UniqueName: \"kubernetes.io/projected/20a93003-13be-468a-aeb8-bfd0bcb86dbd-kube-api-access-txvjb\") pod \"tuned-tk5ng\" (UID: \"20a93003-13be-468a-aeb8-bfd0bcb86dbd\") " pod="openshift-cluster-node-tuning-operator/tuned-tk5ng"
May 25 11:17:37.117698 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.117678    2576 reflector.go:430] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
May 25 11:17:37.118170 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.118150    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-56vfx\" (UniqueName: \"kubernetes.io/projected/a2602414-6828-4ead-beff-a1e1215f9851-kube-api-access-56vfx\") pod \"multus-qjgcp\" (UID: \"a2602414-6828-4ead-beff-a1e1215f9851\") " pod="openshift-multus/multus-qjgcp"
May 25 11:17:37.118848 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.118823    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqd7v\" (UniqueName: \"kubernetes.io/projected/12b45353-8444-41fc-a8ed-fb691764eb63-kube-api-access-rqd7v\") pod \"ovnkube-node-h6b5t\" (UID: \"12b45353-8444-41fc-a8ed-fb691764eb63\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6b5t"
May 25 11:17:37.118848 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.118840    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lp9kp\" (UniqueName: \"kubernetes.io/projected/ff50c150-45a9-41c6-be52-886f2c960e85-kube-api-access-lp9kp\") pod \"network-metrics-daemon-5859j\" (UID: \"ff50c150-45a9-41c6-be52-886f2c960e85\") " pod="openshift-multus/network-metrics-daemon-5859j"
May 25 11:17:37.119073 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.119055    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vsrnp\" (UniqueName: \"kubernetes.io/projected/6d2e36b9-21f2-4f1b-91ee-4c6348cded20-kube-api-access-vsrnp\") pod \"iptables-alerter-qpwv2\" (UID: \"6d2e36b9-21f2-4f1b-91ee-4c6348cded20\") " pod="openshift-network-operator/iptables-alerter-qpwv2"
May 25 11:17:37.120746 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.120729    2576 reflector.go:430] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
May 25 11:17:37.207005 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.206976    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/cd7634d4-8eb9-42a5-9e01-7d94645afbd3-serviceca\") pod \"node-ca-nkcf5\" (UID: \"cd7634d4-8eb9-42a5-9e01-7d94645afbd3\") " pod="openshift-image-registry/node-ca-nkcf5"
May 25 11:17:37.207005 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.207008    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/4ab64dbc-4ab2-4199-bf57-698ec5539507-hosts-file\") pod \"node-resolver-xn288\" (UID: \"4ab64dbc-4ab2-4199-bf57-698ec5539507\") " pod="openshift-dns/node-resolver-xn288"
May 25 11:17:37.207231 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.207033    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cd7634d4-8eb9-42a5-9e01-7d94645afbd3-host\") pod \"node-ca-nkcf5\" (UID: \"cd7634d4-8eb9-42a5-9e01-7d94645afbd3\") " pod="openshift-image-registry/node-ca-nkcf5"
May 25 11:17:37.207231 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.207050    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-55bzk\" (UniqueName: \"kubernetes.io/projected/cd7634d4-8eb9-42a5-9e01-7d94645afbd3-kube-api-access-55bzk\") pod \"node-ca-nkcf5\" (UID: \"cd7634d4-8eb9-42a5-9e01-7d94645afbd3\") " pod="openshift-image-registry/node-ca-nkcf5"
May 25 11:17:37.207231 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.207068    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-djb6n\" (UniqueName: \"kubernetes.io/projected/4ab64dbc-4ab2-4199-bf57-698ec5539507-kube-api-access-djb6n\") pod \"node-resolver-xn288\" (UID: \"4ab64dbc-4ab2-4199-bf57-698ec5539507\") " pod="openshift-dns/node-resolver-xn288"
May 25 11:17:37.207231 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.207110    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cd7634d4-8eb9-42a5-9e01-7d94645afbd3-host\") pod \"node-ca-nkcf5\" (UID: \"cd7634d4-8eb9-42a5-9e01-7d94645afbd3\") " pod="openshift-image-registry/node-ca-nkcf5"
May 25 11:17:37.207231 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.207153    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ab64dbc-4ab2-4199-bf57-698ec5539507-tmp-dir\") pod \"node-resolver-xn288\" (UID: \"4ab64dbc-4ab2-4199-bf57-698ec5539507\") " pod="openshift-dns/node-resolver-xn288"
May 25 11:17:37.207231 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.207111    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/4ab64dbc-4ab2-4199-bf57-698ec5539507-hosts-file\") pod \"node-resolver-xn288\" (UID: \"4ab64dbc-4ab2-4199-bf57-698ec5539507\") " pod="openshift-dns/node-resolver-xn288"
May 25 11:17:37.207551 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.207491    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/cd7634d4-8eb9-42a5-9e01-7d94645afbd3-serviceca\") pod \"node-ca-nkcf5\" (UID: \"cd7634d4-8eb9-42a5-9e01-7d94645afbd3\") " pod="openshift-image-registry/node-ca-nkcf5"
May 25 11:17:37.207551 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.207493    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ab64dbc-4ab2-4199-bf57-698ec5539507-tmp-dir\") pod \"node-resolver-xn288\" (UID: \"4ab64dbc-4ab2-4199-bf57-698ec5539507\") " pod="openshift-dns/node-resolver-xn288"
May 25 11:17:37.218769 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.218704    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-djb6n\" (UniqueName: \"kubernetes.io/projected/4ab64dbc-4ab2-4199-bf57-698ec5539507-kube-api-access-djb6n\") pod \"node-resolver-xn288\" (UID: \"4ab64dbc-4ab2-4199-bf57-698ec5539507\") " pod="openshift-dns/node-resolver-xn288"
May 25 11:17:37.220615 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.220591    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-55bzk\" (UniqueName: \"kubernetes.io/projected/cd7634d4-8eb9-42a5-9e01-7d94645afbd3-kube-api-access-55bzk\") pod \"node-ca-nkcf5\" (UID: \"cd7634d4-8eb9-42a5-9e01-7d94645afbd3\") " pod="openshift-image-registry/node-ca-nkcf5"
May 25 11:17:37.283468 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.283431    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jrgpj"
May 25 11:17:37.294269 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.294236    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-dgspq"
May 25 11:17:37.304949 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.304921    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-qpwv2"
May 25 11:17:37.310071 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.310051    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-h6b5t"
May 25 11:17:37.315631 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.315609    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-4vc6k"
May 25 11:17:37.323156 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.323136    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-qjgcp"
May 25 11:17:37.329699 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.329677    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-tk5ng"
May 25 11:17:37.335901 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.335882    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-xn288"
May 25 11:17:37.342427 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.342403    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-nkcf5"
May 25 11:17:37.609186 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.609086    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ff50c150-45a9-41c6-be52-886f2c960e85-metrics-certs\") pod \"network-metrics-daemon-5859j\" (UID: \"ff50c150-45a9-41c6-be52-886f2c960e85\") " pod="openshift-multus/network-metrics-daemon-5859j"
May 25 11:17:37.609367 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:17:37.609224    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 11:17:37.609367 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:17:37.609322    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ff50c150-45a9-41c6-be52-886f2c960e85-metrics-certs podName:ff50c150-45a9-41c6-be52-886f2c960e85 nodeName:}" failed. No retries permitted until 2026-05-25 11:17:38.609302187 +0000 UTC m=+4.127033621 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ff50c150-45a9-41c6-be52-886f2c960e85-metrics-certs") pod "network-metrics-daemon-5859j" (UID: "ff50c150-45a9-41c6-be52-886f2c960e85") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 11:17:37.696933 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:37.696901    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ae22b47_9a06_49bb_a618_43df38833826.slice/crio-5c9b65f6cdcf8630647c4e8eab6536440574af797098e0c4aa5ea288394484b8 WatchSource:0}: Error finding container 5c9b65f6cdcf8630647c4e8eab6536440574af797098e0c4aa5ea288394484b8: Status 404 returned error can't find the container with id 5c9b65f6cdcf8630647c4e8eab6536440574af797098e0c4aa5ea288394484b8
May 25 11:17:37.698290 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:37.698243    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ab64dbc_4ab2_4199_bf57_698ec5539507.slice/crio-d486f32d847f8f5c14a524f0cfcc90e619da58d185f8a1711732864155f9ebb3 WatchSource:0}: Error finding container d486f32d847f8f5c14a524f0cfcc90e619da58d185f8a1711732864155f9ebb3: Status 404 returned error can't find the container with id d486f32d847f8f5c14a524f0cfcc90e619da58d185f8a1711732864155f9ebb3
May 25 11:17:37.699405 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:37.699294    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2602414_6828_4ead_beff_a1e1215f9851.slice/crio-2b536a04c4152314a14f9cc80186c497d550f7de201210915b0dab003ff64331 WatchSource:0}: Error finding container 2b536a04c4152314a14f9cc80186c497d550f7de201210915b0dab003ff64331: Status 404 returned error can't find the container with id 2b536a04c4152314a14f9cc80186c497d550f7de201210915b0dab003ff64331
May 25 11:17:37.703304 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:37.703263    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddeccfe91_b8ff_47cc_92b2_d6f19d19ebec.slice/crio-6a083d6feb918b438a3fd602597ab61ef62fb230f52271725c1a8bfb292b6bf1 WatchSource:0}: Error finding container 6a083d6feb918b438a3fd602597ab61ef62fb230f52271725c1a8bfb292b6bf1: Status 404 returned error can't find the container with id 6a083d6feb918b438a3fd602597ab61ef62fb230f52271725c1a8bfb292b6bf1
May 25 11:17:37.703715 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:37.703692    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d2e36b9_21f2_4f1b_91ee_4c6348cded20.slice/crio-9cdb67c1deef03a2990e8008c7f63b50d61c403bc73118a76db6d5551fa1b7af WatchSource:0}: Error finding container 9cdb67c1deef03a2990e8008c7f63b50d61c403bc73118a76db6d5551fa1b7af: Status 404 returned error can't find the container with id 9cdb67c1deef03a2990e8008c7f63b50d61c403bc73118a76db6d5551fa1b7af
May 25 11:17:37.704754 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:37.704647    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd7634d4_8eb9_42a5_9e01_7d94645afbd3.slice/crio-fd5abb187d345c56a2ff3b69f4de91ae8194d3d5c02695222b00a164020fd2b8 WatchSource:0}: Error finding container fd5abb187d345c56a2ff3b69f4de91ae8194d3d5c02695222b00a164020fd2b8: Status 404 returned error can't find the container with id fd5abb187d345c56a2ff3b69f4de91ae8194d3d5c02695222b00a164020fd2b8
May 25 11:17:37.705848 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:37.705825    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac23d02e_4fdd_4cfe_a16a_0958e2a7b62e.slice/crio-2f89fff2b6b2b19a172d12c2ba66d00b8618b0380e71f12d0e66b15ab2172610 WatchSource:0}: Error finding container 2f89fff2b6b2b19a172d12c2ba66d00b8618b0380e71f12d0e66b15ab2172610: Status 404 returned error can't find the container with id 2f89fff2b6b2b19a172d12c2ba66d00b8618b0380e71f12d0e66b15ab2172610
May 25 11:17:37.707428 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:37.707410    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20a93003_13be_468a_aeb8_bfd0bcb86dbd.slice/crio-0f72d442a024673cdaadf635b11ff3b20876a32ddc74af4d0e8fec32bfce1ca3 WatchSource:0}: Error finding container 0f72d442a024673cdaadf635b11ff3b20876a32ddc74af4d0e8fec32bfce1ca3: Status 404 returned error can't find the container with id 0f72d442a024673cdaadf635b11ff3b20876a32ddc74af4d0e8fec32bfce1ca3
May 25 11:17:37.708677 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:17:37.708581    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12b45353_8444_41fc_a8ed_fb691764eb63.slice/crio-e6f088926ceeaa3edb755fce4f0a1166a4a980356b4f86059df9dfe756c54c06 WatchSource:0}: Error finding container e6f088926ceeaa3edb755fce4f0a1166a4a980356b4f86059df9dfe756c54c06: Status 404 returned error can't find the container with id e6f088926ceeaa3edb755fce4f0a1166a4a980356b4f86059df9dfe756c54c06
May 25 11:17:37.709495 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:37.709476    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-w5fnt\" (UniqueName: \"kubernetes.io/projected/e77e329e-0279-4bf7-9210-7c8226f6a091-kube-api-access-w5fnt\") pod \"network-check-target-xmvkz\" (UID: \"e77e329e-0279-4bf7-9210-7c8226f6a091\") " pod="openshift-network-diagnostics/network-check-target-xmvkz"
May 25 11:17:37.709705 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:17:37.709632    2576 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 25 11:17:37.709705 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:17:37.709650    2576 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 25 11:17:37.709705 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:17:37.709662    2576 projected.go:194] Error preparing data for projected volume kube-api-access-w5fnt for pod openshift-network-diagnostics/network-check-target-xmvkz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 11:17:37.709848 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:17:37.709732    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e77e329e-0279-4bf7-9210-7c8226f6a091-kube-api-access-w5fnt podName:e77e329e-0279-4bf7-9210-7c8226f6a091 nodeName:}" failed. No retries permitted until 2026-05-25 11:17:38.709712772 +0000 UTC m=+4.227444221 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-w5fnt" (UniqueName: "kubernetes.io/projected/e77e329e-0279-4bf7-9210-7c8226f6a091-kube-api-access-w5fnt") pod "network-check-target-xmvkz" (UID: "e77e329e-0279-4bf7-9210-7c8226f6a091") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 11:17:38.031741 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:38.031706    2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-05-24 11:12:36 +0000 UTC" deadline="2027-12-14 00:56:39.265784806 +0000 UTC"
May 25 11:17:38.031741 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:38.031740    2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13621h39m1.234048653s"
May 25 11:17:38.105161 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:38.105127    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-nkcf5" event={"ID":"cd7634d4-8eb9-42a5-9e01-7d94645afbd3","Type":"ContainerStarted","Data":"fd5abb187d345c56a2ff3b69f4de91ae8194d3d5c02695222b00a164020fd2b8"}
May 25 11:17:38.106423 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:38.106394    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-qpwv2" event={"ID":"6d2e36b9-21f2-4f1b-91ee-4c6348cded20","Type":"ContainerStarted","Data":"9cdb67c1deef03a2990e8008c7f63b50d61c403bc73118a76db6d5551fa1b7af"}
May 25 11:17:38.110415 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:38.110362    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-qjgcp" event={"ID":"a2602414-6828-4ead-beff-a1e1215f9851","Type":"ContainerStarted","Data":"2b536a04c4152314a14f9cc80186c497d550f7de201210915b0dab003ff64331"}
May 25 11:17:38.113538 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:38.113484    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-xn288" event={"ID":"4ab64dbc-4ab2-4199-bf57-698ec5539507","Type":"ContainerStarted","Data":"d486f32d847f8f5c14a524f0cfcc90e619da58d185f8a1711732864155f9ebb3"}
May 25 11:17:38.114696 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:38.114673    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jrgpj" event={"ID":"5ae22b47-9a06-49bb-a618-43df38833826","Type":"ContainerStarted","Data":"5c9b65f6cdcf8630647c4e8eab6536440574af797098e0c4aa5ea288394484b8"}
May 25 11:17:38.115958 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:38.115934    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-h6b5t" event={"ID":"12b45353-8444-41fc-a8ed-fb691764eb63","Type":"ContainerStarted","Data":"e6f088926ceeaa3edb755fce4f0a1166a4a980356b4f86059df9dfe756c54c06"}
May 25 11:17:38.119158 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:38.119062    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-tk5ng" event={"ID":"20a93003-13be-468a-aeb8-bfd0bcb86dbd","Type":"ContainerStarted","Data":"0f72d442a024673cdaadf635b11ff3b20876a32ddc74af4d0e8fec32bfce1ca3"}
May 25 11:17:38.120662 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:38.120618    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dgspq" event={"ID":"ac23d02e-4fdd-4cfe-a16a-0958e2a7b62e","Type":"ContainerStarted","Data":"2f89fff2b6b2b19a172d12c2ba66d00b8618b0380e71f12d0e66b15ab2172610"}
May 25 11:17:38.121766 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:38.121721    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-4vc6k" event={"ID":"deccfe91-b8ff-47cc-92b2-d6f19d19ebec","Type":"ContainerStarted","Data":"6a083d6feb918b438a3fd602597ab61ef62fb230f52271725c1a8bfb292b6bf1"}
May 25 11:17:38.125332 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:38.124701    2576 generic.go:358] "Generic (PLEG): container finished" podID="67ca951ca4e4e1d2d1949fbdcb1b1f35" containerID="54281ea2f970aa3fe7728e0d0efc6c3f71185ce7885179b410b04d878ca83112" exitCode=0
May 25 11:17:38.125332 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:38.124767    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-254.ec2.internal" event={"ID":"67ca951ca4e4e1d2d1949fbdcb1b1f35","Type":"ContainerDied","Data":"54281ea2f970aa3fe7728e0d0efc6c3f71185ce7885179b410b04d878ca83112"}
May 25 11:17:38.135980 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:38.135955    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-254.ec2.internal" event={"ID":"11df30495fdea7c5dad53acc2aaa0a94","Type":"ContainerStarted","Data":"370d12233097516d50918d217d98e4c935ed2a0d7a22860adcf12326592a8f77"}
May 25 11:17:38.616236 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:38.615927    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ff50c150-45a9-41c6-be52-886f2c960e85-metrics-certs\") pod \"network-metrics-daemon-5859j\" (UID: \"ff50c150-45a9-41c6-be52-886f2c960e85\") " pod="openshift-multus/network-metrics-daemon-5859j"
May 25 11:17:38.616404 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:17:38.616339    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 11:17:38.616471 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:17:38.616406    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ff50c150-45a9-41c6-be52-886f2c960e85-metrics-certs podName:ff50c150-45a9-41c6-be52-886f2c960e85 nodeName:}" failed. No retries permitted until 2026-05-25 11:17:40.61638712 +0000 UTC m=+6.134118559 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ff50c150-45a9-41c6-be52-886f2c960e85-metrics-certs") pod "network-metrics-daemon-5859j" (UID: "ff50c150-45a9-41c6-be52-886f2c960e85") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 11:17:38.717353 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:38.717316    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-w5fnt\" (UniqueName: \"kubernetes.io/projected/e77e329e-0279-4bf7-9210-7c8226f6a091-kube-api-access-w5fnt\") pod \"network-check-target-xmvkz\" (UID: \"e77e329e-0279-4bf7-9210-7c8226f6a091\") " pod="openshift-network-diagnostics/network-check-target-xmvkz"
May 25 11:17:38.717531 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:17:38.717508    2576 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 25 11:17:38.717531 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:17:38.717529    2576 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 25 11:17:38.717638 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:17:38.717541    2576 projected.go:194] Error preparing data for projected volume kube-api-access-w5fnt for pod openshift-network-diagnostics/network-check-target-xmvkz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 11:17:38.717638 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:17:38.717601    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e77e329e-0279-4bf7-9210-7c8226f6a091-kube-api-access-w5fnt podName:e77e329e-0279-4bf7-9210-7c8226f6a091 nodeName:}" failed. No retries permitted until 2026-05-25 11:17:40.717582692 +0000 UTC m=+6.235314138 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-w5fnt" (UniqueName: "kubernetes.io/projected/e77e329e-0279-4bf7-9210-7c8226f6a091-kube-api-access-w5fnt") pod "network-check-target-xmvkz" (UID: "e77e329e-0279-4bf7-9210-7c8226f6a091") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 11:17:39.093805 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:39.093722    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5859j"
May 25 11:17:39.094224 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:17:39.093850    2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5859j" podUID="ff50c150-45a9-41c6-be52-886f2c960e85"
May 25 11:17:39.096738 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:39.093653    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xmvkz"
May 25 11:17:39.096738 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:17:39.096516    2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xmvkz" podUID="e77e329e-0279-4bf7-9210-7c8226f6a091"
May 25 11:17:39.153429 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:39.153377    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-254.ec2.internal" event={"ID":"67ca951ca4e4e1d2d1949fbdcb1b1f35","Type":"ContainerStarted","Data":"8b29b60480c218b3768a9abb6042d0622c622d9a45c35967c8305b27f7980f7c"}
May 25 11:17:39.167669 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:39.167614    2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-254.ec2.internal" podStartSLOduration=3.167598094 podStartE2EDuration="3.167598094s" podCreationTimestamp="2026-05-25 11:17:36 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 11:17:38.175525069 +0000 UTC m=+3.693256730" watchObservedRunningTime="2026-05-25 11:17:39.167598094 +0000 UTC m=+4.685329550"
May 25 11:17:39.167800 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:39.167702    2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-254.ec2.internal" podStartSLOduration=3.167696207 podStartE2EDuration="3.167696207s" podCreationTimestamp="2026-05-25 11:17:36 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 11:17:39.166887516 +0000 UTC m=+4.684618973" watchObservedRunningTime="2026-05-25 11:17:39.167696207 +0000 UTC m=+4.685427672"
May 25 11:17:40.632824 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:40.632742    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ff50c150-45a9-41c6-be52-886f2c960e85-metrics-certs\") pod \"network-metrics-daemon-5859j\" (UID: \"ff50c150-45a9-41c6-be52-886f2c960e85\") " pod="openshift-multus/network-metrics-daemon-5859j"
May 25 11:17:40.633251 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:17:40.632939    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 11:17:40.633251 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:17:40.633015    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ff50c150-45a9-41c6-be52-886f2c960e85-metrics-certs podName:ff50c150-45a9-41c6-be52-886f2c960e85 nodeName:}" failed. No retries permitted until 2026-05-25 11:17:44.632996383 +0000 UTC m=+10.150727820 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ff50c150-45a9-41c6-be52-886f2c960e85-metrics-certs") pod "network-metrics-daemon-5859j" (UID: "ff50c150-45a9-41c6-be52-886f2c960e85") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 11:17:40.733673 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:40.733638    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-w5fnt\" (UniqueName: \"kubernetes.io/projected/e77e329e-0279-4bf7-9210-7c8226f6a091-kube-api-access-w5fnt\") pod \"network-check-target-xmvkz\" (UID: \"e77e329e-0279-4bf7-9210-7c8226f6a091\") " pod="openshift-network-diagnostics/network-check-target-xmvkz"
May 25 11:17:40.734304 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:17:40.733822    2576 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 25 11:17:40.734304 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:17:40.733852    2576 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 25 11:17:40.734304 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:17:40.733867    2576 projected.go:194] Error preparing data for projected volume kube-api-access-w5fnt for pod openshift-network-diagnostics/network-check-target-xmvkz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 11:17:40.734304 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:17:40.733928    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e77e329e-0279-4bf7-9210-7c8226f6a091-kube-api-access-w5fnt podName:e77e329e-0279-4bf7-9210-7c8226f6a091 nodeName:}" failed. No retries permitted until 2026-05-25 11:17:44.733908987 +0000 UTC m=+10.251640433 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-w5fnt" (UniqueName: "kubernetes.io/projected/e77e329e-0279-4bf7-9210-7c8226f6a091-kube-api-access-w5fnt") pod "network-check-target-xmvkz" (UID: "e77e329e-0279-4bf7-9210-7c8226f6a091") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 11:17:41.094912 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:41.094837    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xmvkz"
May 25 11:17:41.095055 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:41.094853    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5859j"
May 25 11:17:41.095055 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:17:41.094979    2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xmvkz" podUID="e77e329e-0279-4bf7-9210-7c8226f6a091"
May 25 11:17:41.095931 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:17:41.095900    2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5859j" podUID="ff50c150-45a9-41c6-be52-886f2c960e85"
May 25 11:17:42.056088 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:42.056042    2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-bx7s6"]
May 25 11:17:42.071437 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:42.071405    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bx7s6"
May 25 11:17:42.071591 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:17:42.071487    2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bx7s6" podUID="dc85864b-c44e-4ff8-b9c5-04b7ab66aeb1"
May 25 11:17:42.145575 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:42.145487    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/dc85864b-c44e-4ff8-b9c5-04b7ab66aeb1-dbus\") pod \"global-pull-secret-syncer-bx7s6\" (UID: \"dc85864b-c44e-4ff8-b9c5-04b7ab66aeb1\") " pod="kube-system/global-pull-secret-syncer-bx7s6"
May 25 11:17:42.145739 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:42.145605    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/dc85864b-c44e-4ff8-b9c5-04b7ab66aeb1-kubelet-config\") pod \"global-pull-secret-syncer-bx7s6\" (UID: \"dc85864b-c44e-4ff8-b9c5-04b7ab66aeb1\") " pod="kube-system/global-pull-secret-syncer-bx7s6"
May 25 11:17:42.145739 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:42.145639    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dc85864b-c44e-4ff8-b9c5-04b7ab66aeb1-original-pull-secret\") pod \"global-pull-secret-syncer-bx7s6\" (UID: \"dc85864b-c44e-4ff8-b9c5-04b7ab66aeb1\") " pod="kube-system/global-pull-secret-syncer-bx7s6"
May 25 11:17:42.246691 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:42.246657    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/dc85864b-c44e-4ff8-b9c5-04b7ab66aeb1-kubelet-config\") pod \"global-pull-secret-syncer-bx7s6\" (UID: \"dc85864b-c44e-4ff8-b9c5-04b7ab66aeb1\") " pod="kube-system/global-pull-secret-syncer-bx7s6"
May 25 11:17:42.246853 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:42.246705    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dc85864b-c44e-4ff8-b9c5-04b7ab66aeb1-original-pull-secret\") pod \"global-pull-secret-syncer-bx7s6\" (UID: \"dc85864b-c44e-4ff8-b9c5-04b7ab66aeb1\") " pod="kube-system/global-pull-secret-syncer-bx7s6"
May 25 11:17:42.246853 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:42.246737    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/dc85864b-c44e-4ff8-b9c5-04b7ab66aeb1-dbus\") pod \"global-pull-secret-syncer-bx7s6\" (UID: \"dc85864b-c44e-4ff8-b9c5-04b7ab66aeb1\") " pod="kube-system/global-pull-secret-syncer-bx7s6"
May 25 11:17:42.247029 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:42.247009    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/dc85864b-c44e-4ff8-b9c5-04b7ab66aeb1-dbus\") pod \"global-pull-secret-syncer-bx7s6\" (UID: \"dc85864b-c44e-4ff8-b9c5-04b7ab66aeb1\") " pod="kube-system/global-pull-secret-syncer-bx7s6"
May 25 11:17:42.247156 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:42.247114    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/dc85864b-c44e-4ff8-b9c5-04b7ab66aeb1-kubelet-config\") pod \"global-pull-secret-syncer-bx7s6\" (UID: \"dc85864b-c44e-4ff8-b9c5-04b7ab66aeb1\") " pod="kube-system/global-pull-secret-syncer-bx7s6"
May 25 11:17:42.247214 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:17:42.247155    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
May 25 11:17:42.247257 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:17:42.247230    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc85864b-c44e-4ff8-b9c5-04b7ab66aeb1-original-pull-secret podName:dc85864b-c44e-4ff8-b9c5-04b7ab66aeb1 nodeName:}" failed. No retries permitted until 2026-05-25 11:17:42.747209327 +0000 UTC m=+8.264940760 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/dc85864b-c44e-4ff8-b9c5-04b7ab66aeb1-original-pull-secret") pod "global-pull-secret-syncer-bx7s6" (UID: "dc85864b-c44e-4ff8-b9c5-04b7ab66aeb1") : object "kube-system"/"original-pull-secret" not registered
May 25 11:17:42.751439 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:42.751354    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dc85864b-c44e-4ff8-b9c5-04b7ab66aeb1-original-pull-secret\") pod \"global-pull-secret-syncer-bx7s6\" (UID: \"dc85864b-c44e-4ff8-b9c5-04b7ab66aeb1\") " pod="kube-system/global-pull-secret-syncer-bx7s6"
May 25 11:17:42.751595 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:17:42.751521    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
May 25 11:17:42.751670 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:17:42.751595    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc85864b-c44e-4ff8-b9c5-04b7ab66aeb1-original-pull-secret podName:dc85864b-c44e-4ff8-b9c5-04b7ab66aeb1 nodeName:}" failed. No retries permitted until 2026-05-25 11:17:43.751575536 +0000 UTC m=+9.269306979 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/dc85864b-c44e-4ff8-b9c5-04b7ab66aeb1-original-pull-secret") pod "global-pull-secret-syncer-bx7s6" (UID: "dc85864b-c44e-4ff8-b9c5-04b7ab66aeb1") : object "kube-system"/"original-pull-secret" not registered
May 25 11:17:43.094310 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:43.094201    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xmvkz"
May 25 11:17:43.095431 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:17:43.095396    2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xmvkz" podUID="e77e329e-0279-4bf7-9210-7c8226f6a091"
May 25 11:17:43.095822 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:43.095803    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5859j"
May 25 11:17:43.095940 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:17:43.095901    2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5859j" podUID="ff50c150-45a9-41c6-be52-886f2c960e85"
May 25 11:17:43.758317 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:43.758265    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dc85864b-c44e-4ff8-b9c5-04b7ab66aeb1-original-pull-secret\") pod \"global-pull-secret-syncer-bx7s6\" (UID: \"dc85864b-c44e-4ff8-b9c5-04b7ab66aeb1\") " pod="kube-system/global-pull-secret-syncer-bx7s6"
May 25 11:17:43.758508 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:17:43.758445    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
May 25 11:17:43.758508 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:17:43.758506    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc85864b-c44e-4ff8-b9c5-04b7ab66aeb1-original-pull-secret podName:dc85864b-c44e-4ff8-b9c5-04b7ab66aeb1 nodeName:}" failed. No retries permitted until 2026-05-25 11:17:45.75848753 +0000 UTC m=+11.276218972 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/dc85864b-c44e-4ff8-b9c5-04b7ab66aeb1-original-pull-secret") pod "global-pull-secret-syncer-bx7s6" (UID: "dc85864b-c44e-4ff8-b9c5-04b7ab66aeb1") : object "kube-system"/"original-pull-secret" not registered
May 25 11:17:44.094314 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:44.094092    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bx7s6"
May 25 11:17:44.094750 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:17:44.094268    2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bx7s6" podUID="dc85864b-c44e-4ff8-b9c5-04b7ab66aeb1"
May 25 11:17:44.665198 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:44.665162    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ff50c150-45a9-41c6-be52-886f2c960e85-metrics-certs\") pod \"network-metrics-daemon-5859j\" (UID: \"ff50c150-45a9-41c6-be52-886f2c960e85\") " pod="openshift-multus/network-metrics-daemon-5859j"
May 25 11:17:44.665401 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:17:44.665317    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 11:17:44.665401 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:17:44.665375    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ff50c150-45a9-41c6-be52-886f2c960e85-metrics-certs podName:ff50c150-45a9-41c6-be52-886f2c960e85 nodeName:}" failed. No retries permitted until 2026-05-25 11:17:52.665357879 +0000 UTC m=+18.183089318 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ff50c150-45a9-41c6-be52-886f2c960e85-metrics-certs") pod "network-metrics-daemon-5859j" (UID: "ff50c150-45a9-41c6-be52-886f2c960e85") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 11:17:44.765845 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:44.765705    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-w5fnt\" (UniqueName: \"kubernetes.io/projected/e77e329e-0279-4bf7-9210-7c8226f6a091-kube-api-access-w5fnt\") pod \"network-check-target-xmvkz\" (UID: \"e77e329e-0279-4bf7-9210-7c8226f6a091\") " pod="openshift-network-diagnostics/network-check-target-xmvkz"
May 25 11:17:44.766005 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:17:44.765907    2576 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 25 11:17:44.766005 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:17:44.765931    2576 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 25 11:17:44.766005 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:17:44.765946    2576 projected.go:194] Error preparing data for projected volume kube-api-access-w5fnt for pod openshift-network-diagnostics/network-check-target-xmvkz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 11:17:44.766204 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:17:44.766012    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e77e329e-0279-4bf7-9210-7c8226f6a091-kube-api-access-w5fnt podName:e77e329e-0279-4bf7-9210-7c8226f6a091 nodeName:}" failed. No retries permitted until 2026-05-25 11:17:52.765982713 +0000 UTC m=+18.283714168 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-w5fnt" (UniqueName: "kubernetes.io/projected/e77e329e-0279-4bf7-9210-7c8226f6a091-kube-api-access-w5fnt") pod "network-check-target-xmvkz" (UID: "e77e329e-0279-4bf7-9210-7c8226f6a091") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 11:17:45.097478 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:45.097341    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xmvkz"
May 25 11:17:45.097478 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:45.097351    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5859j"
May 25 11:17:45.097478 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:17:45.097461    2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xmvkz" podUID="e77e329e-0279-4bf7-9210-7c8226f6a091"
May 25 11:17:45.097989 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:17:45.097538    2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5859j" podUID="ff50c150-45a9-41c6-be52-886f2c960e85"
May 25 11:17:45.775540 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:45.775498    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dc85864b-c44e-4ff8-b9c5-04b7ab66aeb1-original-pull-secret\") pod \"global-pull-secret-syncer-bx7s6\" (UID: \"dc85864b-c44e-4ff8-b9c5-04b7ab66aeb1\") " pod="kube-system/global-pull-secret-syncer-bx7s6"
May 25 11:17:45.775714 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:17:45.775643    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
May 25 11:17:45.775771 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:17:45.775716    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc85864b-c44e-4ff8-b9c5-04b7ab66aeb1-original-pull-secret podName:dc85864b-c44e-4ff8-b9c5-04b7ab66aeb1 nodeName:}" failed. No retries permitted until 2026-05-25 11:17:49.775694682 +0000 UTC m=+15.293426116 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/dc85864b-c44e-4ff8-b9c5-04b7ab66aeb1-original-pull-secret") pod "global-pull-secret-syncer-bx7s6" (UID: "dc85864b-c44e-4ff8-b9c5-04b7ab66aeb1") : object "kube-system"/"original-pull-secret" not registered
May 25 11:17:46.093561 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:46.093494    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bx7s6"
May 25 11:17:46.093700 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:17:46.093611    2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bx7s6" podUID="dc85864b-c44e-4ff8-b9c5-04b7ab66aeb1"
May 25 11:17:47.096982 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:47.096955    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xmvkz"
May 25 11:17:47.096982 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:47.096967    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5859j"
May 25 11:17:47.097491 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:17:47.097062    2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xmvkz" podUID="e77e329e-0279-4bf7-9210-7c8226f6a091"
May 25 11:17:47.097491 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:17:47.097193    2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5859j" podUID="ff50c150-45a9-41c6-be52-886f2c960e85"
May 25 11:17:48.093445 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:48.093412    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bx7s6"
May 25 11:17:48.093636 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:17:48.093548    2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bx7s6" podUID="dc85864b-c44e-4ff8-b9c5-04b7ab66aeb1"
May 25 11:17:49.094396 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:49.094365    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xmvkz"
May 25 11:17:49.094942 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:49.094366    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5859j"
May 25 11:17:49.094942 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:17:49.094468    2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xmvkz" podUID="e77e329e-0279-4bf7-9210-7c8226f6a091"
May 25 11:17:49.094942 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:17:49.094580    2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5859j" podUID="ff50c150-45a9-41c6-be52-886f2c960e85"
May 25 11:17:49.806145 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:49.806105    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dc85864b-c44e-4ff8-b9c5-04b7ab66aeb1-original-pull-secret\") pod \"global-pull-secret-syncer-bx7s6\" (UID: \"dc85864b-c44e-4ff8-b9c5-04b7ab66aeb1\") " pod="kube-system/global-pull-secret-syncer-bx7s6"
May 25 11:17:49.806349 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:17:49.806269    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
May 25 11:17:49.806415 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:17:49.806355    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc85864b-c44e-4ff8-b9c5-04b7ab66aeb1-original-pull-secret podName:dc85864b-c44e-4ff8-b9c5-04b7ab66aeb1 nodeName:}" failed. No retries permitted until 2026-05-25 11:17:57.806338776 +0000 UTC m=+23.324070216 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/dc85864b-c44e-4ff8-b9c5-04b7ab66aeb1-original-pull-secret") pod "global-pull-secret-syncer-bx7s6" (UID: "dc85864b-c44e-4ff8-b9c5-04b7ab66aeb1") : object "kube-system"/"original-pull-secret" not registered
May 25 11:17:50.093796 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:50.093720    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bx7s6"
May 25 11:17:50.093950 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:17:50.093838    2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bx7s6" podUID="dc85864b-c44e-4ff8-b9c5-04b7ab66aeb1"
May 25 11:17:51.094010 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:51.093916    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5859j"
May 25 11:17:51.094476 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:51.093916    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xmvkz"
May 25 11:17:51.094476 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:17:51.094055    2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5859j" podUID="ff50c150-45a9-41c6-be52-886f2c960e85"
May 25 11:17:51.094476 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:17:51.094183    2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xmvkz" podUID="e77e329e-0279-4bf7-9210-7c8226f6a091"
May 25 11:17:52.094286 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:52.094241    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bx7s6"
May 25 11:17:52.094806 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:17:52.094360    2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bx7s6" podUID="dc85864b-c44e-4ff8-b9c5-04b7ab66aeb1"
May 25 11:17:52.731678 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:52.731609    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ff50c150-45a9-41c6-be52-886f2c960e85-metrics-certs\") pod \"network-metrics-daemon-5859j\" (UID: \"ff50c150-45a9-41c6-be52-886f2c960e85\") " pod="openshift-multus/network-metrics-daemon-5859j"
May 25 11:17:52.731862 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:17:52.731727    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 11:17:52.731862 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:17:52.731803    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ff50c150-45a9-41c6-be52-886f2c960e85-metrics-certs podName:ff50c150-45a9-41c6-be52-886f2c960e85 nodeName:}" failed. No retries permitted until 2026-05-25 11:18:08.731782931 +0000 UTC m=+34.249514369 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ff50c150-45a9-41c6-be52-886f2c960e85-metrics-certs") pod "network-metrics-daemon-5859j" (UID: "ff50c150-45a9-41c6-be52-886f2c960e85") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 11:17:52.832528 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:52.832491    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-w5fnt\" (UniqueName: \"kubernetes.io/projected/e77e329e-0279-4bf7-9210-7c8226f6a091-kube-api-access-w5fnt\") pod \"network-check-target-xmvkz\" (UID: \"e77e329e-0279-4bf7-9210-7c8226f6a091\") " pod="openshift-network-diagnostics/network-check-target-xmvkz"
May 25 11:17:52.832694 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:17:52.832657    2576 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 25 11:17:52.832737 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:17:52.832681    2576 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 25 11:17:52.832737 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:17:52.832708    2576 projected.go:194] Error preparing data for projected volume kube-api-access-w5fnt for pod openshift-network-diagnostics/network-check-target-xmvkz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 11:17:52.832817 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:17:52.832757    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e77e329e-0279-4bf7-9210-7c8226f6a091-kube-api-access-w5fnt podName:e77e329e-0279-4bf7-9210-7c8226f6a091 nodeName:}" failed. No retries permitted until 2026-05-25 11:18:08.832743949 +0000 UTC m=+34.350475383 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-w5fnt" (UniqueName: "kubernetes.io/projected/e77e329e-0279-4bf7-9210-7c8226f6a091-kube-api-access-w5fnt") pod "network-check-target-xmvkz" (UID: "e77e329e-0279-4bf7-9210-7c8226f6a091") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 11:17:53.093933 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:53.093898    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xmvkz"
May 25 11:17:53.094114 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:53.093904    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5859j"
May 25 11:17:53.094114 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:17:53.094023    2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xmvkz" podUID="e77e329e-0279-4bf7-9210-7c8226f6a091"
May 25 11:17:53.094208 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:17:53.094142    2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5859j" podUID="ff50c150-45a9-41c6-be52-886f2c960e85"
May 25 11:17:54.094222 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:54.094181    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bx7s6"
May 25 11:17:54.094793 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:17:54.094326    2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bx7s6" podUID="dc85864b-c44e-4ff8-b9c5-04b7ab66aeb1"
May 25 11:17:55.095655 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:55.095627    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xmvkz"
May 25 11:17:55.095655 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:55.095638    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5859j"
May 25 11:17:55.096108 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:17:55.095718    2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xmvkz" podUID="e77e329e-0279-4bf7-9210-7c8226f6a091"
May 25 11:17:55.096108 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:17:55.095832    2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5859j" podUID="ff50c150-45a9-41c6-be52-886f2c960e85"
May 25 11:17:56.093918 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:56.093728    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bx7s6"
May 25 11:17:56.094050 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:17:56.093997    2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bx7s6" podUID="dc85864b-c44e-4ff8-b9c5-04b7ab66aeb1"
May 25 11:17:56.181314 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:56.181267    2576 generic.go:358] "Generic (PLEG): container finished" podID="ac23d02e-4fdd-4cfe-a16a-0958e2a7b62e" containerID="9e54f7f77ada92aec6367c50a2a747697fd90d1ad483dccc98035925702f9870" exitCode=0
May 25 11:17:56.181719 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:56.181344    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dgspq" event={"ID":"ac23d02e-4fdd-4cfe-a16a-0958e2a7b62e","Type":"ContainerDied","Data":"9e54f7f77ada92aec6367c50a2a747697fd90d1ad483dccc98035925702f9870"}
May 25 11:17:56.182595 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:56.182556    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-4vc6k" event={"ID":"deccfe91-b8ff-47cc-92b2-d6f19d19ebec","Type":"ContainerStarted","Data":"67e198616a94080e04905366f469ceb143f09fa04710b55080e749cfff94aa56"}
May 25 11:17:56.183800 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:56.183777    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-nkcf5" event={"ID":"cd7634d4-8eb9-42a5-9e01-7d94645afbd3","Type":"ContainerStarted","Data":"11a4ace52ff1d0825cdcedc6c3bac4bc2a82bb3b9c1975ce27665a584e3b2211"}
May 25 11:17:56.184991 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:56.184966    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-qjgcp" event={"ID":"a2602414-6828-4ead-beff-a1e1215f9851","Type":"ContainerStarted","Data":"07a62dbe32657f4b13db82adf577893f5f3e1a12f9cda6f12846decfc0ba0506"}
May 25 11:17:56.186080 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:56.186062    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-xn288" event={"ID":"4ab64dbc-4ab2-4199-bf57-698ec5539507","Type":"ContainerStarted","Data":"bd75f037d46fb76dcca3f641f63e33bcfdd2556e9462d907b3229403aa3b360e"}
May 25 11:17:56.187132 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:56.187111    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jrgpj" event={"ID":"5ae22b47-9a06-49bb-a618-43df38833826","Type":"ContainerStarted","Data":"1dd46671d046a947cb70be17c28ff53b411f9d1f8ed8d84bfbda637903ddfd5a"}
May 25 11:17:56.189015 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:56.188998    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h6b5t_12b45353-8444-41fc-a8ed-fb691764eb63/ovn-acl-logging/0.log"
May 25 11:17:56.189261 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:56.189241    2576 generic.go:358] "Generic (PLEG): container finished" podID="12b45353-8444-41fc-a8ed-fb691764eb63" containerID="f32ebbf26d795ba4f0db382fa54c98385c6cc1ed74f6b0822c171e29ddb4f4ea" exitCode=1
May 25 11:17:56.189330 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:56.189297    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-h6b5t" event={"ID":"12b45353-8444-41fc-a8ed-fb691764eb63","Type":"ContainerStarted","Data":"8cd29290b262ae2267a6e84d29ce02abdae30c4b0b027cf7df6f966f8b56ae97"}
May 25 11:17:56.189330 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:56.189318    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-h6b5t" event={"ID":"12b45353-8444-41fc-a8ed-fb691764eb63","Type":"ContainerDied","Data":"f32ebbf26d795ba4f0db382fa54c98385c6cc1ed74f6b0822c171e29ddb4f4ea"}
May 25 11:17:56.189330 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:56.189328    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-h6b5t" event={"ID":"12b45353-8444-41fc-a8ed-fb691764eb63","Type":"ContainerStarted","Data":"03dfb1671ee12e13e89d66e3c836a9d9f5ec6cea61b499f39390d2df5039a6ee"}
May 25 11:17:56.190320 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:56.190302    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-tk5ng" event={"ID":"20a93003-13be-468a-aeb8-bfd0bcb86dbd","Type":"ContainerStarted","Data":"e6e4882b0cf38296e2e9f43901e4b0722de8c338e63436fbbaaef0deb30b893b"}
May 25 11:17:56.216776 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:56.216733    2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/konnectivity-agent-4vc6k" podStartSLOduration=3.362386894 podStartE2EDuration="21.216720987s" podCreationTimestamp="2026-05-25 11:17:35 +0000 UTC" firstStartedPulling="2026-05-25 11:17:37.705170798 +0000 UTC m=+3.222902247" lastFinishedPulling="2026-05-25 11:17:55.559504899 +0000 UTC m=+21.077236340" observedRunningTime="2026-05-25 11:17:56.216513031 +0000 UTC m=+21.734244479" watchObservedRunningTime="2026-05-25 11:17:56.216720987 +0000 UTC m=+21.734452442"
May 25 11:17:56.237466 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:56.237417    2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/multus-qjgcp" podStartSLOduration=3.313002936 podStartE2EDuration="21.237402923s" podCreationTimestamp="2026-05-25 11:17:35 +0000 UTC" firstStartedPulling="2026-05-25 11:17:37.701374106 +0000 UTC m=+3.219105540" lastFinishedPulling="2026-05-25 11:17:55.625774087 +0000 UTC m=+21.143505527" observedRunningTime="2026-05-25 11:17:56.237379207 +0000 UTC m=+21.755110663" watchObservedRunningTime="2026-05-25 11:17:56.237402923 +0000 UTC m=+21.755134378"
May 25 11:17:56.255918 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:56.255878    2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-tk5ng" podStartSLOduration=3.341798759 podStartE2EDuration="21.25586535s" podCreationTimestamp="2026-05-25 11:17:35 +0000 UTC" firstStartedPulling="2026-05-25 11:17:37.710012792 +0000 UTC m=+3.227744238" lastFinishedPulling="2026-05-25 11:17:55.624079383 +0000 UTC m=+21.141810829" observedRunningTime="2026-05-25 11:17:56.255299314 +0000 UTC m=+21.773030767" watchObservedRunningTime="2026-05-25 11:17:56.25586535 +0000 UTC m=+21.773596805"
May 25 11:17:56.277775 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:56.277738    2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-dns/node-resolver-xn288" podStartSLOduration=3.35324879 podStartE2EDuration="21.277726689s" podCreationTimestamp="2026-05-25 11:17:35 +0000 UTC" firstStartedPulling="2026-05-25 11:17:37.700319539 +0000 UTC m=+3.218050976" lastFinishedPulling="2026-05-25 11:17:55.624797434 +0000 UTC m=+21.142528875" observedRunningTime="2026-05-25 11:17:56.277233158 +0000 UTC m=+21.794964614" watchObservedRunningTime="2026-05-25 11:17:56.277726689 +0000 UTC m=+21.795458145"
May 25 11:17:56.296441 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:56.296402    2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-image-registry/node-ca-nkcf5" podStartSLOduration=3.444084681 podStartE2EDuration="21.29638956s" podCreationTimestamp="2026-05-25 11:17:35 +0000 UTC" firstStartedPulling="2026-05-25 11:17:37.707210416 +0000 UTC m=+3.224941850" lastFinishedPulling="2026-05-25 11:17:55.559515285 +0000 UTC m=+21.077246729" observedRunningTime="2026-05-25 11:17:56.295681281 +0000 UTC m=+21.813412737" watchObservedRunningTime="2026-05-25 11:17:56.29638956 +0000 UTC m=+21.814121016"
May 25 11:17:57.097002 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:57.096979    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5859j"
May 25 11:17:57.097182 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:57.096978    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xmvkz"
May 25 11:17:57.097182 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:17:57.097081    2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5859j" podUID="ff50c150-45a9-41c6-be52-886f2c960e85"
May 25 11:17:57.097182 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:17:57.097163    2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xmvkz" podUID="e77e329e-0279-4bf7-9210-7c8226f6a091"
May 25 11:17:57.194926 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:57.194905    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h6b5t_12b45353-8444-41fc-a8ed-fb691764eb63/ovn-acl-logging/0.log"
May 25 11:17:57.195366 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:57.195315    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-h6b5t" event={"ID":"12b45353-8444-41fc-a8ed-fb691764eb63","Type":"ContainerStarted","Data":"1dc9272aaa0a3867dd92b359eb94a1a6df9ef1887a060d04cd07d9fc19017678"}
May 25 11:17:57.195366 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:57.195351    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-h6b5t" event={"ID":"12b45353-8444-41fc-a8ed-fb691764eb63","Type":"ContainerStarted","Data":"b7de959ff7b818ccfdb1f2f2317f2eea5b267c361edbc2e312fa1f4a8a35ff9e"}
May 25 11:17:57.195366 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:57.195365    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-h6b5t" event={"ID":"12b45353-8444-41fc-a8ed-fb691764eb63","Type":"ContainerStarted","Data":"53b2a6e7d1c41e7643a1368a837b193e43f5cada1b21d4552a8f6339f66f163f"}
May 25 11:17:57.196999 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:57.196907    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-qpwv2" event={"ID":"6d2e36b9-21f2-4f1b-91ee-4c6348cded20","Type":"ContainerStarted","Data":"9021954a0694810edc59c29f756bbf1f154133b5a21e509729acf70661163dbe"}
May 25 11:17:57.212739 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:57.212693    2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-qpwv2" podStartSLOduration=4.286929644 podStartE2EDuration="22.212678275s" podCreationTimestamp="2026-05-25 11:17:35 +0000 UTC" firstStartedPulling="2026-05-25 11:17:37.706028558 +0000 UTC m=+3.223759996" lastFinishedPulling="2026-05-25 11:17:55.631777193 +0000 UTC m=+21.149508627" observedRunningTime="2026-05-25 11:17:57.21218336 +0000 UTC m=+22.729914816" watchObservedRunningTime="2026-05-25 11:17:57.212678275 +0000 UTC m=+22.730409733"
May 25 11:17:57.267141 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:57.267116    2576 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
May 25 11:17:57.870935 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:57.870900    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dc85864b-c44e-4ff8-b9c5-04b7ab66aeb1-original-pull-secret\") pod \"global-pull-secret-syncer-bx7s6\" (UID: \"dc85864b-c44e-4ff8-b9c5-04b7ab66aeb1\") " pod="kube-system/global-pull-secret-syncer-bx7s6"
May 25 11:17:57.871126 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:17:57.871048    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
May 25 11:17:57.871126 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:17:57.871121    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc85864b-c44e-4ff8-b9c5-04b7ab66aeb1-original-pull-secret podName:dc85864b-c44e-4ff8-b9c5-04b7ab66aeb1 nodeName:}" failed. No retries permitted until 2026-05-25 11:18:13.871099736 +0000 UTC m=+39.388831171 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/dc85864b-c44e-4ff8-b9c5-04b7ab66aeb1-original-pull-secret") pod "global-pull-secret-syncer-bx7s6" (UID: "dc85864b-c44e-4ff8-b9c5-04b7ab66aeb1") : object "kube-system"/"original-pull-secret" not registered
May 25 11:17:58.069649 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:58.069498    2576 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-05-25T11:17:57.267138188Z","UUID":"a5068a20-cbe2-41ed-9985-bd8779116181","Handler":null,"Name":"","Endpoint":""}
May 25 11:17:58.073487 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:58.073459    2576 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
May 25 11:17:58.073647 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:58.073497    2576 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
May 25 11:17:58.094371 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:58.094348    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bx7s6"
May 25 11:17:58.094516 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:17:58.094465    2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bx7s6" podUID="dc85864b-c44e-4ff8-b9c5-04b7ab66aeb1"
May 25 11:17:58.200061 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:58.200023    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jrgpj" event={"ID":"5ae22b47-9a06-49bb-a618-43df38833826","Type":"ContainerStarted","Data":"c6a1292e6e174a570347c3dc7a35ffa942d0a33a5db55f6682d70ea1857e36b9"}
May 25 11:17:59.093673 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:59.093641    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xmvkz"
May 25 11:17:59.093849 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:17:59.093760    2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xmvkz" podUID="e77e329e-0279-4bf7-9210-7c8226f6a091"
May 25 11:17:59.093849 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:59.093795    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5859j"
May 25 11:17:59.093978 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:17:59.093918    2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5859j" podUID="ff50c150-45a9-41c6-be52-886f2c960e85"
May 25 11:17:59.204242 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:59.204213    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h6b5t_12b45353-8444-41fc-a8ed-fb691764eb63/ovn-acl-logging/0.log"
May 25 11:17:59.204763 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:59.204634    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-h6b5t" event={"ID":"12b45353-8444-41fc-a8ed-fb691764eb63","Type":"ContainerStarted","Data":"7e580dc10370e44e1f52bb893f7ca17309ea77ae853fb62e50f581d185de5ce3"}
May 25 11:17:59.206662 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:59.206637    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jrgpj" event={"ID":"5ae22b47-9a06-49bb-a618-43df38833826","Type":"ContainerStarted","Data":"dec66a5c990ed406cd321a7233ee464493910b9cdd917d2d9ed0a21ee497804f"}
May 25 11:17:59.229765 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:17:59.229724    2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jrgpj" podStartSLOduration=3.517304555 podStartE2EDuration="24.229712491s" podCreationTimestamp="2026-05-25 11:17:35 +0000 UTC" firstStartedPulling="2026-05-25 11:17:37.699718359 +0000 UTC m=+3.217449796" lastFinishedPulling="2026-05-25 11:17:58.412126286 +0000 UTC m=+23.929857732" observedRunningTime="2026-05-25 11:17:59.228585514 +0000 UTC m=+24.746316971" watchObservedRunningTime="2026-05-25 11:17:59.229712491 +0000 UTC m=+24.747443947"
May 25 11:18:00.093527 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:18:00.093496    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bx7s6"
May 25 11:18:00.093686 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:18:00.093591    2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bx7s6" podUID="dc85864b-c44e-4ff8-b9c5-04b7ab66aeb1"
May 25 11:18:00.596983 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:18:00.596932    2576 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-4vc6k"
May 25 11:18:00.597691 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:18:00.597670    2576 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-4vc6k"
May 25 11:18:01.097002 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:18:01.096774    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xmvkz"
May 25 11:18:01.097189 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:18:01.096774    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5859j"
May 25 11:18:01.097189 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:18:01.097037    2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xmvkz" podUID="e77e329e-0279-4bf7-9210-7c8226f6a091"
May 25 11:18:01.097189 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:18:01.097153    2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5859j" podUID="ff50c150-45a9-41c6-be52-886f2c960e85"
May 25 11:18:01.211256 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:18:01.211224    2576 generic.go:358] "Generic (PLEG): container finished" podID="ac23d02e-4fdd-4cfe-a16a-0958e2a7b62e" containerID="34a4471fc90e30ea78e8cabaf49f63f35329ce8c02ebd199f9d3e4dcad283fa8" exitCode=0
May 25 11:18:01.211399 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:18:01.211318    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dgspq" event={"ID":"ac23d02e-4fdd-4cfe-a16a-0958e2a7b62e","Type":"ContainerDied","Data":"34a4471fc90e30ea78e8cabaf49f63f35329ce8c02ebd199f9d3e4dcad283fa8"}
May 25 11:18:01.214397 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:18:01.214379    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h6b5t_12b45353-8444-41fc-a8ed-fb691764eb63/ovn-acl-logging/0.log"
May 25 11:18:01.214741 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:18:01.214721    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-h6b5t" event={"ID":"12b45353-8444-41fc-a8ed-fb691764eb63","Type":"ContainerStarted","Data":"c167fee436366754d662449cf667fde7fc431658ed8aa432f883c01da15a17b8"}
May 25 11:18:01.215061 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:18:01.215044    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-4vc6k"
May 25 11:18:01.215142 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:18:01.215127    2576 scope.go:117] "RemoveContainer" containerID="f32ebbf26d795ba4f0db382fa54c98385c6cc1ed74f6b0822c171e29ddb4f4ea"
May 25 11:18:01.215483 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:18:01.215468    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-4vc6k"
May 25 11:18:02.093557 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:18:02.093528    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bx7s6"
May 25 11:18:02.093967 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:18:02.093623    2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bx7s6" podUID="dc85864b-c44e-4ff8-b9c5-04b7ab66aeb1"
May 25 11:18:02.218940 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:18:02.218911    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h6b5t_12b45353-8444-41fc-a8ed-fb691764eb63/ovn-acl-logging/0.log"
May 25 11:18:02.219337 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:18:02.219311    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-h6b5t" event={"ID":"12b45353-8444-41fc-a8ed-fb691764eb63","Type":"ContainerStarted","Data":"fc5df39d92dea05d3d42f770691ebe1e2d5d25cdec5d2f0e590721258fa2712a"}
May 25 11:18:02.219627 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:18:02.219602    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-h6b5t"
May 25 11:18:02.219705 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:18:02.219654    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-h6b5t"
May 25 11:18:02.219705 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:18:02.219668    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-h6b5t"
May 25 11:18:02.235028 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:18:02.235005    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-h6b5t"
May 25 11:18:02.235860 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:18:02.235844    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-h6b5t"
May 25 11:18:02.252923 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:18:02.252884    2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-h6b5t" podStartSLOduration=9.315815324 podStartE2EDuration="27.252872401s" podCreationTimestamp="2026-05-25 11:17:35 +0000 UTC" firstStartedPulling="2026-05-25 11:17:37.711819253 +0000 UTC m=+3.229550699" lastFinishedPulling="2026-05-25 11:17:55.648876342 +0000 UTC m=+21.166607776" observedRunningTime="2026-05-25 11:18:02.252269686 +0000 UTC m=+27.770001143" watchObservedRunningTime="2026-05-25 11:18:02.252872401 +0000 UTC m=+27.770603870"
May 25 11:18:02.660441 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:18:02.660184    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-bx7s6"]
May 25 11:18:02.660564 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:18:02.660466    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bx7s6"
May 25 11:18:02.660607 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:18:02.660557    2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bx7s6" podUID="dc85864b-c44e-4ff8-b9c5-04b7ab66aeb1"
May 25 11:18:02.669883 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:18:02.669854    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-xmvkz"]
May 25 11:18:02.670032 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:18:02.670011    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xmvkz"
May 25 11:18:02.670117 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:18:02.670101    2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xmvkz" podUID="e77e329e-0279-4bf7-9210-7c8226f6a091"
May 25 11:18:02.670526 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:18:02.670504    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-5859j"]
May 25 11:18:02.670620 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:18:02.670607    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5859j"
May 25 11:18:02.670732 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:18:02.670709    2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5859j" podUID="ff50c150-45a9-41c6-be52-886f2c960e85"
May 25 11:18:03.222403 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:18:03.222371    2576 generic.go:358] "Generic (PLEG): container finished" podID="ac23d02e-4fdd-4cfe-a16a-0958e2a7b62e" containerID="dc66145f7490f86456b9966d4ac1cce0f77af0dcd8a3c0aae6d25b0725faf9f1" exitCode=0
May 25 11:18:03.223059 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:18:03.222443    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dgspq" event={"ID":"ac23d02e-4fdd-4cfe-a16a-0958e2a7b62e","Type":"ContainerDied","Data":"dc66145f7490f86456b9966d4ac1cce0f77af0dcd8a3c0aae6d25b0725faf9f1"}
May 25 11:18:04.094104 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:18:04.094068    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xmvkz"
May 25 11:18:04.094248 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:18:04.094068    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bx7s6"
May 25 11:18:04.094248 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:18:04.094173    2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xmvkz" podUID="e77e329e-0279-4bf7-9210-7c8226f6a091"
May 25 11:18:04.094347 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:18:04.094245    2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bx7s6" podUID="dc85864b-c44e-4ff8-b9c5-04b7ab66aeb1"
May 25 11:18:04.094347 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:18:04.094068    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5859j"
May 25 11:18:04.094410 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:18:04.094349    2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5859j" podUID="ff50c150-45a9-41c6-be52-886f2c960e85"
May 25 11:18:05.227832 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:18:05.227799    2576 generic.go:358] "Generic (PLEG): container finished" podID="ac23d02e-4fdd-4cfe-a16a-0958e2a7b62e" containerID="d76b0229a68e15c12e6bd26070ca0561a8f0cc70348312f08826734d93933fd1" exitCode=0
May 25 11:18:05.228417 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:18:05.227855    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dgspq" event={"ID":"ac23d02e-4fdd-4cfe-a16a-0958e2a7b62e","Type":"ContainerDied","Data":"d76b0229a68e15c12e6bd26070ca0561a8f0cc70348312f08826734d93933fd1"}
May 25 11:18:06.094320 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:18:06.094095    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5859j"
May 25 11:18:06.094492 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:18:06.094106    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bx7s6"
May 25 11:18:06.094492 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:18:06.094132    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xmvkz"
May 25 11:18:06.094492 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:18:06.094468    2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5859j" podUID="ff50c150-45a9-41c6-be52-886f2c960e85"
May 25 11:18:06.094624 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:18:06.094535    2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bx7s6" podUID="dc85864b-c44e-4ff8-b9c5-04b7ab66aeb1"
May 25 11:18:06.095323 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:18:06.094925    2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xmvkz" podUID="e77e329e-0279-4bf7-9210-7c8226f6a091"
May 25 11:18:08.093771 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:18:08.093739    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xmvkz"
May 25 11:18:08.094224 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:18:08.093739    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bx7s6"
May 25 11:18:08.094224 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:18:08.093855    2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xmvkz" podUID="e77e329e-0279-4bf7-9210-7c8226f6a091"
May 25 11:18:08.094224 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:18:08.093939    2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bx7s6" podUID="dc85864b-c44e-4ff8-b9c5-04b7ab66aeb1"
May 25 11:18:08.094224 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:18:08.093744    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5859j"
May 25 11:18:08.094224 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:18:08.094040    2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5859j" podUID="ff50c150-45a9-41c6-be52-886f2c960e85"
May 25 11:18:08.758906 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:18:08.758873    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ff50c150-45a9-41c6-be52-886f2c960e85-metrics-certs\") pod \"network-metrics-daemon-5859j\" (UID: \"ff50c150-45a9-41c6-be52-886f2c960e85\") " pod="openshift-multus/network-metrics-daemon-5859j"
May 25 11:18:08.759163 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:18:08.759014    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 11:18:08.759163 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:18:08.759088    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ff50c150-45a9-41c6-be52-886f2c960e85-metrics-certs podName:ff50c150-45a9-41c6-be52-886f2c960e85 nodeName:}" failed. No retries permitted until 2026-05-25 11:18:40.759069534 +0000 UTC m=+66.276800973 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ff50c150-45a9-41c6-be52-886f2c960e85-metrics-certs") pod "network-metrics-daemon-5859j" (UID: "ff50c150-45a9-41c6-be52-886f2c960e85") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 11:18:08.787039 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:18:08.787013    2576 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-132-254.ec2.internal" event="NodeReady"
May 25 11:18:08.787191 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:18:08.787172    2576 kubelet_node_status.go:550] "Fast updating node status as it just became ready"
May 25 11:18:08.850348 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:18:08.850313    2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-8lxgp"]
May 25 11:18:08.859619 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:18:08.859587    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-w5fnt\" (UniqueName: \"kubernetes.io/projected/e77e329e-0279-4bf7-9210-7c8226f6a091-kube-api-access-w5fnt\") pod \"network-check-target-xmvkz\" (UID: \"e77e329e-0279-4bf7-9210-7c8226f6a091\") " pod="openshift-network-diagnostics/network-check-target-xmvkz"
May 25 11:18:08.859772 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:18:08.859740    2576 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 25 11:18:08.859772 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:18:08.859758    2576 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 25 11:18:08.859772 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:18:08.859768    2576 projected.go:194] Error preparing data for projected volume kube-api-access-w5fnt for pod openshift-network-diagnostics/network-check-target-xmvkz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 11:18:08.859935 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:18:08.859821    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e77e329e-0279-4bf7-9210-7c8226f6a091-kube-api-access-w5fnt podName:e77e329e-0279-4bf7-9210-7c8226f6a091 nodeName:}" failed. No retries permitted until 2026-05-25 11:18:40.859802624 +0000 UTC m=+66.377534066 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-w5fnt" (UniqueName: "kubernetes.io/projected/e77e329e-0279-4bf7-9210-7c8226f6a091-kube-api-access-w5fnt") pod "network-check-target-xmvkz" (UID: "e77e329e-0279-4bf7-9210-7c8226f6a091") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 11:18:08.873623 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:18:08.873596    2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-cpqmb"]
May 25 11:18:08.873826 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:18:08.873805    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-8lxgp"
May 25 11:18:08.876666 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:18:08.876393    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
May 25 11:18:08.876666 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:18:08.876423    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
May 25 11:18:08.876834 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:18:08.876675    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-t8nfn\""
May 25 11:18:08.887529 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:18:08.887503    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-cpqmb"]
May 25 11:18:08.887611 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:18:08.887533    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-8lxgp"]
May 25 11:18:08.887611 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:18:08.887597    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-cpqmb"
May 25 11:18:08.889711 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:18:08.889693    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-mjf4k\""
May 25 11:18:08.889814 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:18:08.889768    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
May 25 11:18:08.890028 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:18:08.890003    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
May 25 11:18:08.890265 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:18:08.890250    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
May 25 11:18:08.960192 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:18:08.960064    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/62aaed5e-f12a-4833-acbc-4ea8bfe20fb1-config-volume\") pod \"dns-default-8lxgp\" (UID: \"62aaed5e-f12a-4833-acbc-4ea8bfe20fb1\") " pod="openshift-dns/dns-default-8lxgp"
May 25 11:18:08.960192 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:18:08.960129    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62aaed5e-f12a-4833-acbc-4ea8bfe20fb1-tmp-dir\") pod \"dns-default-8lxgp\" (UID: \"62aaed5e-f12a-4833-acbc-4ea8bfe20fb1\") " pod="openshift-dns/dns-default-8lxgp"
May 25 11:18:08.960451 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:18:08.960299    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vsvwh\" (UniqueName: \"kubernetes.io/projected/62aaed5e-f12a-4833-acbc-4ea8bfe20fb1-kube-api-access-vsvwh\") pod \"dns-default-8lxgp\" (UID: \"62aaed5e-f12a-4833-acbc-4ea8bfe20fb1\") " pod="openshift-dns/dns-default-8lxgp"
May 25 11:18:08.960451 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:18:08.960344    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/62aaed5e-f12a-4833-acbc-4ea8bfe20fb1-metrics-tls\") pod \"dns-default-8lxgp\" (UID: \"62aaed5e-f12a-4833-acbc-4ea8bfe20fb1\") " pod="openshift-dns/dns-default-8lxgp"
May 25 11:18:09.061657 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:18:09.061575    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bnwpf\" (UniqueName: \"kubernetes.io/projected/c25259af-1ae3-465a-a802-2dd6fe637e06-kube-api-access-bnwpf\") pod \"ingress-canary-cpqmb\" (UID: \"c25259af-1ae3-465a-a802-2dd6fe637e06\") " pod="openshift-ingress-canary/ingress-canary-cpqmb"
May 25 11:18:09.061657 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:18:09.061620    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/62aaed5e-f12a-4833-acbc-4ea8bfe20fb1-config-volume\") pod \"dns-default-8lxgp\" (UID: \"62aaed5e-f12a-4833-acbc-4ea8bfe20fb1\") " pod="openshift-dns/dns-default-8lxgp"
May 25 11:18:09.061873 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:18:09.061689    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62aaed5e-f12a-4833-acbc-4ea8bfe20fb1-tmp-dir\") pod \"dns-default-8lxgp\" (UID: \"62aaed5e-f12a-4833-acbc-4ea8bfe20fb1\") " pod="openshift-dns/dns-default-8lxgp"
May 25 11:18:09.061873 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:18:09.061765    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c25259af-1ae3-465a-a802-2dd6fe637e06-cert\") pod \"ingress-canary-cpqmb\" (UID: \"c25259af-1ae3-465a-a802-2dd6fe637e06\") " pod="openshift-ingress-canary/ingress-canary-cpqmb"
May 25 11:18:09.061979 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:18:09.061869    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-vsvwh\" (UniqueName: \"kubernetes.io/projected/62aaed5e-f12a-4833-acbc-4ea8bfe20fb1-kube-api-access-vsvwh\") pod \"dns-default-8lxgp\" (UID: \"62aaed5e-f12a-4833-acbc-4ea8bfe20fb1\") " pod="openshift-dns/dns-default-8lxgp"
May 25 11:18:09.061979 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:18:09.061915    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/62aaed5e-f12a-4833-acbc-4ea8bfe20fb1-metrics-tls\") pod \"dns-default-8lxgp\" (UID: \"62aaed5e-f12a-4833-acbc-4ea8bfe20fb1\") " pod="openshift-dns/dns-default-8lxgp"
May 25 11:18:09.062091 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:18:09.062030    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62aaed5e-f12a-4833-acbc-4ea8bfe20fb1-tmp-dir\") pod \"dns-default-8lxgp\" (UID: \"62aaed5e-f12a-4833-acbc-4ea8bfe20fb1\") " pod="openshift-dns/dns-default-8lxgp"
May 25 11:18:09.062091 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:18:09.062046    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 25 11:18:09.062188 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:18:09.062106    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62aaed5e-f12a-4833-acbc-4ea8bfe20fb1-metrics-tls podName:62aaed5e-f12a-4833-acbc-4ea8bfe20fb1 nodeName:}" failed. No retries permitted until 2026-05-25 11:18:09.562087356 +0000 UTC m=+35.079818803 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/62aaed5e-f12a-4833-acbc-4ea8bfe20fb1-metrics-tls") pod "dns-default-8lxgp" (UID: "62aaed5e-f12a-4833-acbc-4ea8bfe20fb1") : secret "dns-default-metrics-tls" not found
May 25 11:18:09.062249 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:18:09.062198    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/62aaed5e-f12a-4833-acbc-4ea8bfe20fb1-config-volume\") pod \"dns-default-8lxgp\" (UID: \"62aaed5e-f12a-4833-acbc-4ea8bfe20fb1\") " pod="openshift-dns/dns-default-8lxgp"
May 25 11:18:09.083650 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:18:09.083624    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vsvwh\" (UniqueName: \"kubernetes.io/projected/62aaed5e-f12a-4833-acbc-4ea8bfe20fb1-kube-api-access-vsvwh\") pod \"dns-default-8lxgp\" (UID: \"62aaed5e-f12a-4833-acbc-4ea8bfe20fb1\") " pod="openshift-dns/dns-default-8lxgp"
May 25 11:18:09.163242 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:18:09.163204    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-bnwpf\" (UniqueName: \"kubernetes.io/projected/c25259af-1ae3-465a-a802-2dd6fe637e06-kube-api-access-bnwpf\") pod \"ingress-canary-cpqmb\" (UID: \"c25259af-1ae3-465a-a802-2dd6fe637e06\") " pod="openshift-ingress-canary/ingress-canary-cpqmb"
May 25 11:18:09.163648 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:18:09.163311    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c25259af-1ae3-465a-a802-2dd6fe637e06-cert\") pod \"ingress-canary-cpqmb\" (UID: \"c25259af-1ae3-465a-a802-2dd6fe637e06\") " pod="openshift-ingress-canary/ingress-canary-cpqmb"
May 25 11:18:09.163648 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:18:09.163428    2576 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 25 11:18:09.163648 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:18:09.163501    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c25259af-1ae3-465a-a802-2dd6fe637e06-cert podName:c25259af-1ae3-465a-a802-2dd6fe637e06 nodeName:}" failed. No retries permitted until 2026-05-25 11:18:09.663480754 +0000 UTC m=+35.181212203 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/c25259af-1ae3-465a-a802-2dd6fe637e06-cert") pod "ingress-canary-cpqmb" (UID: "c25259af-1ae3-465a-a802-2dd6fe637e06") : secret "canary-serving-cert" not found
May 25 11:18:09.172262 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:18:09.172233    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnwpf\" (UniqueName: \"kubernetes.io/projected/c25259af-1ae3-465a-a802-2dd6fe637e06-kube-api-access-bnwpf\") pod \"ingress-canary-cpqmb\" (UID: \"c25259af-1ae3-465a-a802-2dd6fe637e06\") " pod="openshift-ingress-canary/ingress-canary-cpqmb"
May 25 11:18:09.566940 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:18:09.566903    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/62aaed5e-f12a-4833-acbc-4ea8bfe20fb1-metrics-tls\") pod \"dns-default-8lxgp\" (UID: \"62aaed5e-f12a-4833-acbc-4ea8bfe20fb1\") " pod="openshift-dns/dns-default-8lxgp"
May 25 11:18:09.567150 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:18:09.567030    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 25 11:18:09.567150 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:18:09.567100    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62aaed5e-f12a-4833-acbc-4ea8bfe20fb1-metrics-tls podName:62aaed5e-f12a-4833-acbc-4ea8bfe20fb1 nodeName:}" failed. No retries permitted until 2026-05-25 11:18:10.56707926 +0000 UTC m=+36.084810707 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/62aaed5e-f12a-4833-acbc-4ea8bfe20fb1-metrics-tls") pod "dns-default-8lxgp" (UID: "62aaed5e-f12a-4833-acbc-4ea8bfe20fb1") : secret "dns-default-metrics-tls" not found
May 25 11:18:09.667800 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:18:09.667762    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c25259af-1ae3-465a-a802-2dd6fe637e06-cert\") pod \"ingress-canary-cpqmb\" (UID: \"c25259af-1ae3-465a-a802-2dd6fe637e06\") " pod="openshift-ingress-canary/ingress-canary-cpqmb"
May 25 11:18:09.667961 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:18:09.667927    2576 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 25 11:18:09.668024 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:18:09.668002    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c25259af-1ae3-465a-a802-2dd6fe637e06-cert podName:c25259af-1ae3-465a-a802-2dd6fe637e06 nodeName:}" failed. No retries permitted until 2026-05-25 11:18:10.667982209 +0000 UTC m=+36.185713658 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/c25259af-1ae3-465a-a802-2dd6fe637e06-cert") pod "ingress-canary-cpqmb" (UID: "c25259af-1ae3-465a-a802-2dd6fe637e06") : secret "canary-serving-cert" not found
May 25 11:18:10.094013 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:18:10.093980    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xmvkz"
May 25 11:18:10.094013 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:18:10.094003    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bx7s6"
May 25 11:18:10.094352 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:18:10.094330    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5859j"
May 25 11:18:10.097312 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:18:10.097214    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
May 25 11:18:10.097312 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:18:10.097248    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-bl7fp\""
May 25 11:18:10.097312 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:18:10.097266    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
May 25 11:18:10.097550 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:18:10.097324    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
May 25 11:18:10.097550 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:18:10.097248    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-j5m5s\""
May 25 11:18:10.097550 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:18:10.097221    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
May 25 11:18:10.575048 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:18:10.575010    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/62aaed5e-f12a-4833-acbc-4ea8bfe20fb1-metrics-tls\") pod \"dns-default-8lxgp\" (UID: \"62aaed5e-f12a-4833-acbc-4ea8bfe20fb1\") " pod="openshift-dns/dns-default-8lxgp"
May 25 11:18:10.575478 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:18:10.575161    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 25 11:18:10.575478 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:18:10.575253    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62aaed5e-f12a-4833-acbc-4ea8bfe20fb1-metrics-tls podName:62aaed5e-f12a-4833-acbc-4ea8bfe20fb1 nodeName:}" failed. No retries permitted until 2026-05-25 11:18:12.575236099 +0000 UTC m=+38.092967533 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/62aaed5e-f12a-4833-acbc-4ea8bfe20fb1-metrics-tls") pod "dns-default-8lxgp" (UID: "62aaed5e-f12a-4833-acbc-4ea8bfe20fb1") : secret "dns-default-metrics-tls" not found
May 25 11:18:10.675461 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:18:10.675435    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c25259af-1ae3-465a-a802-2dd6fe637e06-cert\") pod \"ingress-canary-cpqmb\" (UID: \"c25259af-1ae3-465a-a802-2dd6fe637e06\") " pod="openshift-ingress-canary/ingress-canary-cpqmb"
May 25 11:18:10.675582 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:18:10.675530    2576 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 25 11:18:10.675632 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:18:10.675589    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c25259af-1ae3-465a-a802-2dd6fe637e06-cert podName:c25259af-1ae3-465a-a802-2dd6fe637e06 nodeName:}" failed. No retries permitted until 2026-05-25 11:18:12.675574232 +0000 UTC m=+38.193305674 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/c25259af-1ae3-465a-a802-2dd6fe637e06-cert") pod "ingress-canary-cpqmb" (UID: "c25259af-1ae3-465a-a802-2dd6fe637e06") : secret "canary-serving-cert" not found
May 25 11:18:11.242270 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:18:11.242236    2576 generic.go:358] "Generic (PLEG): container finished" podID="ac23d02e-4fdd-4cfe-a16a-0958e2a7b62e" containerID="a73c331160abbaaabb22117c6d9115daac29e8dc9085d7a8eddc8bfe6bcb07c5" exitCode=0
May 25 11:18:11.242450 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:18:11.242314    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dgspq" event={"ID":"ac23d02e-4fdd-4cfe-a16a-0958e2a7b62e","Type":"ContainerDied","Data":"a73c331160abbaaabb22117c6d9115daac29e8dc9085d7a8eddc8bfe6bcb07c5"}
May 25 11:18:12.246360 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:18:12.246332    2576 generic.go:358] "Generic (PLEG): container finished" podID="ac23d02e-4fdd-4cfe-a16a-0958e2a7b62e" containerID="d5f03091141e73af852bfe811ce3f40975f4da6971cadfa36de40d378ad468e3" exitCode=0
May 25 11:18:12.246665 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:18:12.246371    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dgspq" event={"ID":"ac23d02e-4fdd-4cfe-a16a-0958e2a7b62e","Type":"ContainerDied","Data":"d5f03091141e73af852bfe811ce3f40975f4da6971cadfa36de40d378ad468e3"}
May 25 11:18:12.591231 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:18:12.591016    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/62aaed5e-f12a-4833-acbc-4ea8bfe20fb1-metrics-tls\") pod \"dns-default-8lxgp\" (UID: \"62aaed5e-f12a-4833-acbc-4ea8bfe20fb1\") " pod="openshift-dns/dns-default-8lxgp"
May 25 11:18:12.591231 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:18:12.591146    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 25 11:18:12.591419 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:18:12.591245    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62aaed5e-f12a-4833-acbc-4ea8bfe20fb1-metrics-tls podName:62aaed5e-f12a-4833-acbc-4ea8bfe20fb1 nodeName:}" failed. No retries permitted until 2026-05-25 11:18:16.591229878 +0000 UTC m=+42.108961317 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/62aaed5e-f12a-4833-acbc-4ea8bfe20fb1-metrics-tls") pod "dns-default-8lxgp" (UID: "62aaed5e-f12a-4833-acbc-4ea8bfe20fb1") : secret "dns-default-metrics-tls" not found
May 25 11:18:12.692288 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:18:12.692238    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c25259af-1ae3-465a-a802-2dd6fe637e06-cert\") pod \"ingress-canary-cpqmb\" (UID: \"c25259af-1ae3-465a-a802-2dd6fe637e06\") " pod="openshift-ingress-canary/ingress-canary-cpqmb"
May 25 11:18:12.692390 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:18:12.692360    2576 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 25 11:18:12.692427 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:18:12.692408    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c25259af-1ae3-465a-a802-2dd6fe637e06-cert podName:c25259af-1ae3-465a-a802-2dd6fe637e06 nodeName:}" failed. No retries permitted until 2026-05-25 11:18:16.692394827 +0000 UTC m=+42.210126261 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/c25259af-1ae3-465a-a802-2dd6fe637e06-cert") pod "ingress-canary-cpqmb" (UID: "c25259af-1ae3-465a-a802-2dd6fe637e06") : secret "canary-serving-cert" not found
May 25 11:18:13.250561 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:18:13.250523    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dgspq" event={"ID":"ac23d02e-4fdd-4cfe-a16a-0958e2a7b62e","Type":"ContainerStarted","Data":"66257f9f9475fe798b49ce98e082139028d0a1cde571bc9b18e30b0b1b014d4c"}
May 25 11:18:13.284687 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:18:13.284634    2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-dgspq" podStartSLOduration=5.321404656 podStartE2EDuration="38.284618593s" podCreationTimestamp="2026-05-25 11:17:35 +0000 UTC" firstStartedPulling="2026-05-25 11:17:37.708999255 +0000 UTC m=+3.226730699" lastFinishedPulling="2026-05-25 11:18:10.672213203 +0000 UTC m=+36.189944636" observedRunningTime="2026-05-25 11:18:13.28437912 +0000 UTC m=+38.802110576" watchObservedRunningTime="2026-05-25 11:18:13.284618593 +0000 UTC m=+38.802350050"
May 25 11:18:13.901444 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:18:13.901416    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dc85864b-c44e-4ff8-b9c5-04b7ab66aeb1-original-pull-secret\") pod \"global-pull-secret-syncer-bx7s6\" (UID: \"dc85864b-c44e-4ff8-b9c5-04b7ab66aeb1\") " pod="kube-system/global-pull-secret-syncer-bx7s6"
May 25 11:18:13.904644 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:18:13.904625    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dc85864b-c44e-4ff8-b9c5-04b7ab66aeb1-original-pull-secret\") pod \"global-pull-secret-syncer-bx7s6\" (UID: \"dc85864b-c44e-4ff8-b9c5-04b7ab66aeb1\") " pod="kube-system/global-pull-secret-syncer-bx7s6"
May 25 11:18:14.012468 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:18:14.012444    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bx7s6"
May 25 11:18:14.194347 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:18:14.194318    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-bx7s6"]
May 25 11:18:14.197691 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:18:14.197647    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc85864b_c44e_4ff8_b9c5_04b7ab66aeb1.slice/crio-556d9a51846b894e2730afa8d1ed45f29f76b9e42c246e1334e3ddba82311667 WatchSource:0}: Error finding container 556d9a51846b894e2730afa8d1ed45f29f76b9e42c246e1334e3ddba82311667: Status 404 returned error can't find the container with id 556d9a51846b894e2730afa8d1ed45f29f76b9e42c246e1334e3ddba82311667
May 25 11:18:14.253186 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:18:14.253158    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-bx7s6" event={"ID":"dc85864b-c44e-4ff8-b9c5-04b7ab66aeb1","Type":"ContainerStarted","Data":"556d9a51846b894e2730afa8d1ed45f29f76b9e42c246e1334e3ddba82311667"}
May 25 11:18:16.618865 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:18:16.618824    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/62aaed5e-f12a-4833-acbc-4ea8bfe20fb1-metrics-tls\") pod \"dns-default-8lxgp\" (UID: \"62aaed5e-f12a-4833-acbc-4ea8bfe20fb1\") " pod="openshift-dns/dns-default-8lxgp"
May 25 11:18:16.619260 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:18:16.618979    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 25 11:18:16.619260 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:18:16.619057    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62aaed5e-f12a-4833-acbc-4ea8bfe20fb1-metrics-tls podName:62aaed5e-f12a-4833-acbc-4ea8bfe20fb1 nodeName:}" failed. No retries permitted until 2026-05-25 11:18:24.619037618 +0000 UTC m=+50.136769053 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/62aaed5e-f12a-4833-acbc-4ea8bfe20fb1-metrics-tls") pod "dns-default-8lxgp" (UID: "62aaed5e-f12a-4833-acbc-4ea8bfe20fb1") : secret "dns-default-metrics-tls" not found
May 25 11:18:16.720112 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:18:16.720075    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c25259af-1ae3-465a-a802-2dd6fe637e06-cert\") pod \"ingress-canary-cpqmb\" (UID: \"c25259af-1ae3-465a-a802-2dd6fe637e06\") " pod="openshift-ingress-canary/ingress-canary-cpqmb"
May 25 11:18:16.720267 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:18:16.720231    2576 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 25 11:18:16.720339 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:18:16.720315    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c25259af-1ae3-465a-a802-2dd6fe637e06-cert podName:c25259af-1ae3-465a-a802-2dd6fe637e06 nodeName:}" failed. No retries permitted until 2026-05-25 11:18:24.720298765 +0000 UTC m=+50.238030198 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/c25259af-1ae3-465a-a802-2dd6fe637e06-cert") pod "ingress-canary-cpqmb" (UID: "c25259af-1ae3-465a-a802-2dd6fe637e06") : secret "canary-serving-cert" not found
May 25 11:18:18.262503 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:18:18.262461    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-bx7s6" event={"ID":"dc85864b-c44e-4ff8-b9c5-04b7ab66aeb1","Type":"ContainerStarted","Data":"b0965b76e3517234629c08b79496c0cc703da7d59b2273b607bcd0eb5bb377c2"}
May 25 11:18:24.676876 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:18:24.676837    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/62aaed5e-f12a-4833-acbc-4ea8bfe20fb1-metrics-tls\") pod \"dns-default-8lxgp\" (UID: \"62aaed5e-f12a-4833-acbc-4ea8bfe20fb1\") " pod="openshift-dns/dns-default-8lxgp"
May 25 11:18:24.677306 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:18:24.676953    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 25 11:18:24.677306 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:18:24.677006    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62aaed5e-f12a-4833-acbc-4ea8bfe20fb1-metrics-tls podName:62aaed5e-f12a-4833-acbc-4ea8bfe20fb1 nodeName:}" failed. No retries permitted until 2026-05-25 11:18:40.676992103 +0000 UTC m=+66.194723542 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/62aaed5e-f12a-4833-acbc-4ea8bfe20fb1-metrics-tls") pod "dns-default-8lxgp" (UID: "62aaed5e-f12a-4833-acbc-4ea8bfe20fb1") : secret "dns-default-metrics-tls" not found
May 25 11:18:24.777508 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:18:24.777474    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c25259af-1ae3-465a-a802-2dd6fe637e06-cert\") pod \"ingress-canary-cpqmb\" (UID: \"c25259af-1ae3-465a-a802-2dd6fe637e06\") " pod="openshift-ingress-canary/ingress-canary-cpqmb"
May 25 11:18:24.777722 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:18:24.777648    2576 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 25 11:18:24.777787 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:18:24.777740    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c25259af-1ae3-465a-a802-2dd6fe637e06-cert podName:c25259af-1ae3-465a-a802-2dd6fe637e06 nodeName:}" failed. No retries permitted until 2026-05-25 11:18:40.777718649 +0000 UTC m=+66.295450093 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/c25259af-1ae3-465a-a802-2dd6fe637e06-cert") pod "ingress-canary-cpqmb" (UID: "c25259af-1ae3-465a-a802-2dd6fe637e06") : secret "canary-serving-cert" not found
May 25 11:18:34.234881 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:18:34.234850    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-h6b5t"
May 25 11:18:34.269540 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:18:34.269495    2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-bx7s6" podStartSLOduration=48.902186188 podStartE2EDuration="52.269482952s" podCreationTimestamp="2026-05-25 11:17:42 +0000 UTC" firstStartedPulling="2026-05-25 11:18:14.199782541 +0000 UTC m=+39.717513978" lastFinishedPulling="2026-05-25 11:18:17.567079305 +0000 UTC m=+43.084810742" observedRunningTime="2026-05-25 11:18:18.286778716 +0000 UTC m=+43.804510173" watchObservedRunningTime="2026-05-25 11:18:34.269482952 +0000 UTC m=+59.787214433"
May 25 11:18:40.682031 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:18:40.681976    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/62aaed5e-f12a-4833-acbc-4ea8bfe20fb1-metrics-tls\") pod \"dns-default-8lxgp\" (UID: \"62aaed5e-f12a-4833-acbc-4ea8bfe20fb1\") " pod="openshift-dns/dns-default-8lxgp"
May 25 11:18:40.682516 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:18:40.682100    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 25 11:18:40.682516 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:18:40.682162    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62aaed5e-f12a-4833-acbc-4ea8bfe20fb1-metrics-tls podName:62aaed5e-f12a-4833-acbc-4ea8bfe20fb1 nodeName:}" failed. No retries permitted until 2026-05-25 11:19:12.682148026 +0000 UTC m=+98.199879460 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/62aaed5e-f12a-4833-acbc-4ea8bfe20fb1-metrics-tls") pod "dns-default-8lxgp" (UID: "62aaed5e-f12a-4833-acbc-4ea8bfe20fb1") : secret "dns-default-metrics-tls" not found
May 25 11:18:40.782897 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:18:40.782868    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c25259af-1ae3-465a-a802-2dd6fe637e06-cert\") pod \"ingress-canary-cpqmb\" (UID: \"c25259af-1ae3-465a-a802-2dd6fe637e06\") " pod="openshift-ingress-canary/ingress-canary-cpqmb"
May 25 11:18:40.782968 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:18:40.782905    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ff50c150-45a9-41c6-be52-886f2c960e85-metrics-certs\") pod \"network-metrics-daemon-5859j\" (UID: \"ff50c150-45a9-41c6-be52-886f2c960e85\") " pod="openshift-multus/network-metrics-daemon-5859j"
May 25 11:18:40.783051 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:18:40.783008    2576 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 25 11:18:40.783209 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:18:40.783091    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c25259af-1ae3-465a-a802-2dd6fe637e06-cert podName:c25259af-1ae3-465a-a802-2dd6fe637e06 nodeName:}" failed. No retries permitted until 2026-05-25 11:19:12.783073868 +0000 UTC m=+98.300805307 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/c25259af-1ae3-465a-a802-2dd6fe637e06-cert") pod "ingress-canary-cpqmb" (UID: "c25259af-1ae3-465a-a802-2dd6fe637e06") : secret "canary-serving-cert" not found
May 25 11:18:40.786557 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:18:40.786535    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
May 25 11:18:40.793266 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:18:40.793247    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
May 25 11:18:40.793332 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:18:40.793316    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ff50c150-45a9-41c6-be52-886f2c960e85-metrics-certs podName:ff50c150-45a9-41c6-be52-886f2c960e85 nodeName:}" failed. No retries permitted until 2026-05-25 11:19:44.793299629 +0000 UTC m=+130.311031075 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ff50c150-45a9-41c6-be52-886f2c960e85-metrics-certs") pod "network-metrics-daemon-5859j" (UID: "ff50c150-45a9-41c6-be52-886f2c960e85") : secret "metrics-daemon-secret" not found
May 25 11:18:40.884053 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:18:40.884029    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-w5fnt\" (UniqueName: \"kubernetes.io/projected/e77e329e-0279-4bf7-9210-7c8226f6a091-kube-api-access-w5fnt\") pod \"network-check-target-xmvkz\" (UID: \"e77e329e-0279-4bf7-9210-7c8226f6a091\") " pod="openshift-network-diagnostics/network-check-target-xmvkz"
May 25 11:18:40.886434 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:18:40.886417    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
May 25 11:18:40.897685 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:18:40.897666    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
May 25 11:18:40.907112 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:18:40.907092    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5fnt\" (UniqueName: \"kubernetes.io/projected/e77e329e-0279-4bf7-9210-7c8226f6a091-kube-api-access-w5fnt\") pod \"network-check-target-xmvkz\" (UID: \"e77e329e-0279-4bf7-9210-7c8226f6a091\") " pod="openshift-network-diagnostics/network-check-target-xmvkz"
May 25 11:18:41.010189 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:18:41.010121    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-j5m5s\""
May 25 11:18:41.016241 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:18:41.016220    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xmvkz"
May 25 11:18:41.142020 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:18:41.141983    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-xmvkz"]
May 25 11:18:41.144881 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:18:41.144847    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode77e329e_0279_4bf7_9210_7c8226f6a091.slice/crio-ab67a636c3af375bfe1242a3151af127fb91183b7269472b446cedc4e2a28bdf WatchSource:0}: Error finding container ab67a636c3af375bfe1242a3151af127fb91183b7269472b446cedc4e2a28bdf: Status 404 returned error can't find the container with id ab67a636c3af375bfe1242a3151af127fb91183b7269472b446cedc4e2a28bdf
May 25 11:18:41.304076 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:18:41.304021    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-xmvkz" event={"ID":"e77e329e-0279-4bf7-9210-7c8226f6a091","Type":"ContainerStarted","Data":"ab67a636c3af375bfe1242a3151af127fb91183b7269472b446cedc4e2a28bdf"}
May 25 11:18:44.311629 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:18:44.311589    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-xmvkz" event={"ID":"e77e329e-0279-4bf7-9210-7c8226f6a091","Type":"ContainerStarted","Data":"6200ba91dba28de8f11185e97792c08068a6df2545f65816d562e8baf652c928"}
May 25 11:18:44.312077 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:18:44.311749    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-xmvkz"
May 25 11:18:44.326674 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:18:44.326620    2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-xmvkz" podStartSLOduration=66.879163435 podStartE2EDuration="1m9.326606015s" podCreationTimestamp="2026-05-25 11:17:35 +0000 UTC" firstStartedPulling="2026-05-25 11:18:41.146631856 +0000 UTC m=+66.664363293" lastFinishedPulling="2026-05-25 11:18:43.59407444 +0000 UTC m=+69.111805873" observedRunningTime="2026-05-25 11:18:44.326348298 +0000 UTC m=+69.844079753" watchObservedRunningTime="2026-05-25 11:18:44.326606015 +0000 UTC m=+69.844337471"
May 25 11:19:12.690162 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:19:12.690130    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/62aaed5e-f12a-4833-acbc-4ea8bfe20fb1-metrics-tls\") pod \"dns-default-8lxgp\" (UID: \"62aaed5e-f12a-4833-acbc-4ea8bfe20fb1\") " pod="openshift-dns/dns-default-8lxgp"
May 25 11:19:12.690588 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:19:12.690235    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 25 11:19:12.690588 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:19:12.690314    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62aaed5e-f12a-4833-acbc-4ea8bfe20fb1-metrics-tls podName:62aaed5e-f12a-4833-acbc-4ea8bfe20fb1 nodeName:}" failed. No retries permitted until 2026-05-25 11:20:16.690298326 +0000 UTC m=+162.208029761 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/62aaed5e-f12a-4833-acbc-4ea8bfe20fb1-metrics-tls") pod "dns-default-8lxgp" (UID: "62aaed5e-f12a-4833-acbc-4ea8bfe20fb1") : secret "dns-default-metrics-tls" not found
May 25 11:19:12.790812 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:19:12.790767    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c25259af-1ae3-465a-a802-2dd6fe637e06-cert\") pod \"ingress-canary-cpqmb\" (UID: \"c25259af-1ae3-465a-a802-2dd6fe637e06\") " pod="openshift-ingress-canary/ingress-canary-cpqmb"
May 25 11:19:12.790950 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:19:12.790908    2576 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 25 11:19:12.790988 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:19:12.790974    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c25259af-1ae3-465a-a802-2dd6fe637e06-cert podName:c25259af-1ae3-465a-a802-2dd6fe637e06 nodeName:}" failed. No retries permitted until 2026-05-25 11:20:16.790957905 +0000 UTC m=+162.308689351 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/c25259af-1ae3-465a-a802-2dd6fe637e06-cert") pod "ingress-canary-cpqmb" (UID: "c25259af-1ae3-465a-a802-2dd6fe637e06") : secret "canary-serving-cert" not found
May 25 11:19:15.316297 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:19:15.316253    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-xmvkz"
May 25 11:19:21.866016 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:19:21.865983    2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-6ddd66b668-q2wsn"]
May 25 11:19:21.869237 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:19:21.869221    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-6ddd66b668-q2wsn"
May 25 11:19:21.871742 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:19:21.871715    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
May 25 11:19:21.871881 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:19:21.871758    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
May 25 11:19:21.871881 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:19:21.871778    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
May 25 11:19:21.871881 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:19:21.871835    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-lkc78\""
May 25 11:19:21.872062 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:19:21.872016    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
May 25 11:19:21.872374 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:19:21.872357    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
May 25 11:19:21.872552 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:19:21.872539    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
May 25 11:19:21.882716 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:19:21.882694    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-6ddd66b668-q2wsn"]
May 25 11:19:21.943830 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:19:21.943805    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/39cfa0b7-b3f2-4927-abb6-996baf27f85f-metrics-certs\") pod \"router-default-6ddd66b668-q2wsn\" (UID: \"39cfa0b7-b3f2-4927-abb6-996baf27f85f\") " pod="openshift-ingress/router-default-6ddd66b668-q2wsn"
May 25 11:19:21.943949 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:19:21.943837    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/39cfa0b7-b3f2-4927-abb6-996baf27f85f-service-ca-bundle\") pod \"router-default-6ddd66b668-q2wsn\" (UID: \"39cfa0b7-b3f2-4927-abb6-996baf27f85f\") " pod="openshift-ingress/router-default-6ddd66b668-q2wsn"
May 25 11:19:21.943949 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:19:21.943863    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bd6rj\" (UniqueName: \"kubernetes.io/projected/39cfa0b7-b3f2-4927-abb6-996baf27f85f-kube-api-access-bd6rj\") pod \"router-default-6ddd66b668-q2wsn\" (UID: \"39cfa0b7-b3f2-4927-abb6-996baf27f85f\") " pod="openshift-ingress/router-default-6ddd66b668-q2wsn"
May 25 11:19:21.943949 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:19:21.943910    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/39cfa0b7-b3f2-4927-abb6-996baf27f85f-stats-auth\") pod \"router-default-6ddd66b668-q2wsn\" (UID: \"39cfa0b7-b3f2-4927-abb6-996baf27f85f\") " pod="openshift-ingress/router-default-6ddd66b668-q2wsn"
May 25 11:19:21.943949 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:19:21.943935    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/39cfa0b7-b3f2-4927-abb6-996baf27f85f-default-certificate\") pod \"router-default-6ddd66b668-q2wsn\" (UID: \"39cfa0b7-b3f2-4927-abb6-996baf27f85f\") " pod="openshift-ingress/router-default-6ddd66b668-q2wsn"
May 25 11:19:21.964005 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:19:21.963977    2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6f78b857d8-gcqd2"]
May 25 11:19:21.966264 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:19:21.966246    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6f78b857d8-gcqd2"
May 25 11:19:21.968873 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:19:21.968844    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
May 25 11:19:21.969077 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:19:21.969060    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-srlsx\""
May 25 11:19:21.969386 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:19:21.969364    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
May 25 11:19:21.969492 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:19:21.969412    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
May 25 11:19:21.978563 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:19:21.978541    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6f78b857d8-gcqd2"]
May 25 11:19:21.978935 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:19:21.978916    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
May 25 11:19:22.044367 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:19:22.044337    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6926af5f-04cd-4fed-bb10-669c46bc0c60-installation-pull-secrets\") pod \"image-registry-6f78b857d8-gcqd2\" (UID: \"6926af5f-04cd-4fed-bb10-669c46bc0c60\") " pod="openshift-image-registry/image-registry-6f78b857d8-gcqd2"
May 25 11:19:22.044479 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:19:22.044385    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-bd6rj\" (UniqueName: \"kubernetes.io/projected/39cfa0b7-b3f2-4927-abb6-996baf27f85f-kube-api-access-bd6rj\") pod \"router-default-6ddd66b668-q2wsn\" (UID: \"39cfa0b7-b3f2-4927-abb6-996baf27f85f\") " pod="openshift-ingress/router-default-6ddd66b668-q2wsn"
May 25 11:19:22.044479 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:19:22.044417    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/39cfa0b7-b3f2-4927-abb6-996baf27f85f-stats-auth\") pod \"router-default-6ddd66b668-q2wsn\" (UID: \"39cfa0b7-b3f2-4927-abb6-996baf27f85f\") " pod="openshift-ingress/router-default-6ddd66b668-q2wsn"
May 25 11:19:22.044479 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:19:22.044444    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/39cfa0b7-b3f2-4927-abb6-996baf27f85f-default-certificate\") pod \"router-default-6ddd66b668-q2wsn\" (UID: \"39cfa0b7-b3f2-4927-abb6-996baf27f85f\") " pod="openshift-ingress/router-default-6ddd66b668-q2wsn"
May 25 11:19:22.044642 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:19:22.044509    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6926af5f-04cd-4fed-bb10-669c46bc0c60-image-registry-private-configuration\") pod \"image-registry-6f78b857d8-gcqd2\" (UID: \"6926af5f-04cd-4fed-bb10-669c46bc0c60\") " pod="openshift-image-registry/image-registry-6f78b857d8-gcqd2"
May 25 11:19:22.044642 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:19:22.044550    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/39cfa0b7-b3f2-4927-abb6-996baf27f85f-metrics-certs\") pod \"router-default-6ddd66b668-q2wsn\" (UID: \"39cfa0b7-b3f2-4927-abb6-996baf27f85f\") " pod="openshift-ingress/router-default-6ddd66b668-q2wsn"
May 25 11:19:22.044642 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:19:22.044580    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6926af5f-04cd-4fed-bb10-669c46bc0c60-registry-certificates\") pod \"image-registry-6f78b857d8-gcqd2\" (UID: \"6926af5f-04cd-4fed-bb10-669c46bc0c60\") " pod="openshift-image-registry/image-registry-6f78b857d8-gcqd2"
May 25 11:19:22.044642 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:19:22.044605    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6926af5f-04cd-4fed-bb10-669c46bc0c60-trusted-ca\") pod \"image-registry-6f78b857d8-gcqd2\" (UID: \"6926af5f-04cd-4fed-bb10-669c46bc0c60\") " pod="openshift-image-registry/image-registry-6f78b857d8-gcqd2"
May 25 11:19:22.044781 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:19:22.044663    2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
May 25 11:19:22.044781 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:19:22.044704    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6926af5f-04cd-4fed-bb10-669c46bc0c60-ca-trust-extracted\") pod \"image-registry-6f78b857d8-gcqd2\" (UID: \"6926af5f-04cd-4fed-bb10-669c46bc0c60\") " pod="openshift-image-registry/image-registry-6f78b857d8-gcqd2"
May 25 11:19:22.044781 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:19:22.044736    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/39cfa0b7-b3f2-4927-abb6-996baf27f85f-metrics-certs podName:39cfa0b7-b3f2-4927-abb6-996baf27f85f nodeName:}" failed. No retries permitted until 2026-05-25 11:19:22.54471455 +0000 UTC m=+108.062445985 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/39cfa0b7-b3f2-4927-abb6-996baf27f85f-metrics-certs") pod "router-default-6ddd66b668-q2wsn" (UID: "39cfa0b7-b3f2-4927-abb6-996baf27f85f") : secret "router-metrics-certs-default" not found
May 25 11:19:22.044916 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:19:22.044803    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6926af5f-04cd-4fed-bb10-669c46bc0c60-bound-sa-token\") pod \"image-registry-6f78b857d8-gcqd2\" (UID: \"6926af5f-04cd-4fed-bb10-669c46bc0c60\") " pod="openshift-image-registry/image-registry-6f78b857d8-gcqd2"
May 25 11:19:22.044916 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:19:22.044827    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4r7f\" (UniqueName: \"kubernetes.io/projected/6926af5f-04cd-4fed-bb10-669c46bc0c60-kube-api-access-d4r7f\") pod \"image-registry-6f78b857d8-gcqd2\" (UID: \"6926af5f-04cd-4fed-bb10-669c46bc0c60\") " pod="openshift-image-registry/image-registry-6f78b857d8-gcqd2"
May 25 11:19:22.044916 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:19:22.044891    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/39cfa0b7-b3f2-4927-abb6-996baf27f85f-service-ca-bundle\") pod \"router-default-6ddd66b668-q2wsn\" (UID: \"39cfa0b7-b3f2-4927-abb6-996baf27f85f\") " pod="openshift-ingress/router-default-6ddd66b668-q2wsn"
May 25 11:19:22.045038 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:19:22.044984    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/39cfa0b7-b3f2-4927-abb6-996baf27f85f-service-ca-bundle podName:39cfa0b7-b3f2-4927-abb6-996baf27f85f nodeName:}" failed. No retries permitted until 2026-05-25 11:19:22.54497122 +0000 UTC m=+108.062702670 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/39cfa0b7-b3f2-4927-abb6-996baf27f85f-service-ca-bundle") pod "router-default-6ddd66b668-q2wsn" (UID: "39cfa0b7-b3f2-4927-abb6-996baf27f85f") : configmap references non-existent config key: service-ca.crt
May 25 11:19:22.045038 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:19:22.045020    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6926af5f-04cd-4fed-bb10-669c46bc0c60-registry-tls\") pod \"image-registry-6f78b857d8-gcqd2\" (UID: \"6926af5f-04cd-4fed-bb10-669c46bc0c60\") " pod="openshift-image-registry/image-registry-6f78b857d8-gcqd2"
May 25 11:19:22.046820 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:19:22.046793    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/39cfa0b7-b3f2-4927-abb6-996baf27f85f-stats-auth\") pod \"router-default-6ddd66b668-q2wsn\" (UID: \"39cfa0b7-b3f2-4927-abb6-996baf27f85f\") " pod="openshift-ingress/router-default-6ddd66b668-q2wsn"
May 25 11:19:22.046901 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:19:22.046827    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/39cfa0b7-b3f2-4927-abb6-996baf27f85f-default-certificate\") pod \"router-default-6ddd66b668-q2wsn\" (UID: \"39cfa0b7-b3f2-4927-abb6-996baf27f85f\") " pod="openshift-ingress/router-default-6ddd66b668-q2wsn"
May 25 11:19:22.059116 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:19:22.059087    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bd6rj\" (UniqueName: \"kubernetes.io/projected/39cfa0b7-b3f2-4927-abb6-996baf27f85f-kube-api-access-bd6rj\") pod \"router-default-6ddd66b668-q2wsn\" (UID: \"39cfa0b7-b3f2-4927-abb6-996baf27f85f\") " pod="openshift-ingress/router-default-6ddd66b668-q2wsn"
May 25 11:19:22.146147 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:19:22.146087    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6926af5f-04cd-4fed-bb10-669c46bc0c60-bound-sa-token\") pod \"image-registry-6f78b857d8-gcqd2\" (UID: \"6926af5f-04cd-4fed-bb10-669c46bc0c60\") " pod="openshift-image-registry/image-registry-6f78b857d8-gcqd2"
May 25 11:19:22.146147 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:19:22.146116    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-d4r7f\" (UniqueName: \"kubernetes.io/projected/6926af5f-04cd-4fed-bb10-669c46bc0c60-kube-api-access-d4r7f\") pod \"image-registry-6f78b857d8-gcqd2\" (UID: \"6926af5f-04cd-4fed-bb10-669c46bc0c60\") " pod="openshift-image-registry/image-registry-6f78b857d8-gcqd2"
May 25 11:19:22.146147 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:19:22.146145    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6926af5f-04cd-4fed-bb10-669c46bc0c60-registry-tls\") pod \"image-registry-6f78b857d8-gcqd2\" (UID: \"6926af5f-04cd-4fed-bb10-669c46bc0c60\") " pod="openshift-image-registry/image-registry-6f78b857d8-gcqd2"
May 25 11:19:22.146374 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:19:22.146160    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6926af5f-04cd-4fed-bb10-669c46bc0c60-installation-pull-secrets\") pod \"image-registry-6f78b857d8-gcqd2\" (UID: \"6926af5f-04cd-4fed-bb10-669c46bc0c60\") " pod="openshift-image-registry/image-registry-6f78b857d8-gcqd2"
May 25 11:19:22.146374 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:19:22.146219    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6926af5f-04cd-4fed-bb10-669c46bc0c60-image-registry-private-configuration\") pod \"image-registry-6f78b857d8-gcqd2\" (UID: \"6926af5f-04cd-4fed-bb10-669c46bc0c60\") " pod="openshift-image-registry/image-registry-6f78b857d8-gcqd2"
May 25 11:19:22.146374 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:19:22.146252    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6926af5f-04cd-4fed-bb10-669c46bc0c60-registry-certificates\") pod \"image-registry-6f78b857d8-gcqd2\" (UID: \"6926af5f-04cd-4fed-bb10-669c46bc0c60\") " pod="openshift-image-registry/image-registry-6f78b857d8-gcqd2"
May 25 11:19:22.146374 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:19:22.146254    2576 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
May 25 11:19:22.146374 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:19:22.146271    2576 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6f78b857d8-gcqd2: secret "image-registry-tls" not found
May 25 11:19:22.146374 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:19:22.146292    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6926af5f-04cd-4fed-bb10-669c46bc0c60-trusted-ca\") pod \"image-registry-6f78b857d8-gcqd2\" (UID: \"6926af5f-04cd-4fed-bb10-669c46bc0c60\") " pod="openshift-image-registry/image-registry-6f78b857d8-gcqd2"
May 25 11:19:22.146374 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:19:22.146320    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6926af5f-04cd-4fed-bb10-669c46bc0c60-ca-trust-extracted\") pod \"image-registry-6f78b857d8-gcqd2\" (UID: \"6926af5f-04cd-4fed-bb10-669c46bc0c60\") " pod="openshift-image-registry/image-registry-6f78b857d8-gcqd2"
May 25 11:19:22.146374 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:19:22.146338    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6926af5f-04cd-4fed-bb10-669c46bc0c60-registry-tls podName:6926af5f-04cd-4fed-bb10-669c46bc0c60 nodeName:}" failed. No retries permitted until 2026-05-25 11:19:22.646319714 +0000 UTC m=+108.164051160 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6926af5f-04cd-4fed-bb10-669c46bc0c60-registry-tls") pod "image-registry-6f78b857d8-gcqd2" (UID: "6926af5f-04cd-4fed-bb10-669c46bc0c60") : secret "image-registry-tls" not found
May 25 11:19:22.146718 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:19:22.146701    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6926af5f-04cd-4fed-bb10-669c46bc0c60-ca-trust-extracted\") pod \"image-registry-6f78b857d8-gcqd2\" (UID: \"6926af5f-04cd-4fed-bb10-669c46bc0c60\") " pod="openshift-image-registry/image-registry-6f78b857d8-gcqd2"
May 25 11:19:22.147026 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:19:22.146999    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6926af5f-04cd-4fed-bb10-669c46bc0c60-registry-certificates\") pod \"image-registry-6f78b857d8-gcqd2\" (UID: \"6926af5f-04cd-4fed-bb10-669c46bc0c60\") " pod="openshift-image-registry/image-registry-6f78b857d8-gcqd2"
May 25 11:19:22.147181 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:19:22.147163    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6926af5f-04cd-4fed-bb10-669c46bc0c60-trusted-ca\") pod \"image-registry-6f78b857d8-gcqd2\" (UID: \"6926af5f-04cd-4fed-bb10-669c46bc0c60\") " pod="openshift-image-registry/image-registry-6f78b857d8-gcqd2"
May 25 11:19:22.148309 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:19:22.148273    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6926af5f-04cd-4fed-bb10-669c46bc0c60-image-registry-private-configuration\") pod \"image-registry-6f78b857d8-gcqd2\" (UID: \"6926af5f-04cd-4fed-bb10-669c46bc0c60\") " pod="openshift-image-registry/image-registry-6f78b857d8-gcqd2"
May 25 11:19:22.148652 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:19:22.148633    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6926af5f-04cd-4fed-bb10-669c46bc0c60-installation-pull-secrets\") pod \"image-registry-6f78b857d8-gcqd2\" (UID: \"6926af5f-04cd-4fed-bb10-669c46bc0c60\") " pod="openshift-image-registry/image-registry-6f78b857d8-gcqd2"
May 25 11:19:22.156326 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:19:22.156307    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6926af5f-04cd-4fed-bb10-669c46bc0c60-bound-sa-token\") pod \"image-registry-6f78b857d8-gcqd2\" (UID: \"6926af5f-04cd-4fed-bb10-669c46bc0c60\") " pod="openshift-image-registry/image-registry-6f78b857d8-gcqd2"
May 25 11:19:22.156412 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:19:22.156328    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4r7f\" (UniqueName: \"kubernetes.io/projected/6926af5f-04cd-4fed-bb10-669c46bc0c60-kube-api-access-d4r7f\") pod \"image-registry-6f78b857d8-gcqd2\" (UID: \"6926af5f-04cd-4fed-bb10-669c46bc0c60\") " pod="openshift-image-registry/image-registry-6f78b857d8-gcqd2"
May 25 11:19:22.549805 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:19:22.549768    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/39cfa0b7-b3f2-4927-abb6-996baf27f85f-service-ca-bundle\") pod \"router-default-6ddd66b668-q2wsn\" (UID: \"39cfa0b7-b3f2-4927-abb6-996baf27f85f\") " pod="openshift-ingress/router-default-6ddd66b668-q2wsn"
May 25 11:19:22.549998 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:19:22.549873    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/39cfa0b7-b3f2-4927-abb6-996baf27f85f-metrics-certs\") pod \"router-default-6ddd66b668-q2wsn\" (UID: \"39cfa0b7-b3f2-4927-abb6-996baf27f85f\") " pod="openshift-ingress/router-default-6ddd66b668-q2wsn"
May 25 11:19:22.549998 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:19:22.549945    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/39cfa0b7-b3f2-4927-abb6-996baf27f85f-service-ca-bundle podName:39cfa0b7-b3f2-4927-abb6-996baf27f85f nodeName:}" failed. No retries permitted until 2026-05-25 11:19:23.549928438 +0000 UTC m=+109.067659878 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/39cfa0b7-b3f2-4927-abb6-996baf27f85f-service-ca-bundle") pod "router-default-6ddd66b668-q2wsn" (UID: "39cfa0b7-b3f2-4927-abb6-996baf27f85f") : configmap references non-existent config key: service-ca.crt
May 25 11:19:22.549998 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:19:22.549980    2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
May 25 11:19:22.550160 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:19:22.550049    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/39cfa0b7-b3f2-4927-abb6-996baf27f85f-metrics-certs podName:39cfa0b7-b3f2-4927-abb6-996baf27f85f nodeName:}" failed. No retries permitted until 2026-05-25 11:19:23.550030537 +0000 UTC m=+109.067761971 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/39cfa0b7-b3f2-4927-abb6-996baf27f85f-metrics-certs") pod "router-default-6ddd66b668-q2wsn" (UID: "39cfa0b7-b3f2-4927-abb6-996baf27f85f") : secret "router-metrics-certs-default" not found
May 25 11:19:22.653640 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:19:22.652033    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6926af5f-04cd-4fed-bb10-669c46bc0c60-registry-tls\") pod \"image-registry-6f78b857d8-gcqd2\" (UID: \"6926af5f-04cd-4fed-bb10-669c46bc0c60\") " pod="openshift-image-registry/image-registry-6f78b857d8-gcqd2"
May 25 11:19:22.653640 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:19:22.652179    2576 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
May 25 11:19:22.653640 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:19:22.652199    2576 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6f78b857d8-gcqd2: secret "image-registry-tls" not found
May 25 11:19:22.653640 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:19:22.652252    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6926af5f-04cd-4fed-bb10-669c46bc0c60-registry-tls podName:6926af5f-04cd-4fed-bb10-669c46bc0c60 nodeName:}" failed. No retries permitted until 2026-05-25 11:19:23.652230426 +0000 UTC m=+109.169961861 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6926af5f-04cd-4fed-bb10-669c46bc0c60-registry-tls") pod "image-registry-6f78b857d8-gcqd2" (UID: "6926af5f-04cd-4fed-bb10-669c46bc0c60") : secret "image-registry-tls" not found
May 25 11:19:23.559056 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:19:23.559012    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/39cfa0b7-b3f2-4927-abb6-996baf27f85f-metrics-certs\") pod \"router-default-6ddd66b668-q2wsn\" (UID: \"39cfa0b7-b3f2-4927-abb6-996baf27f85f\") " pod="openshift-ingress/router-default-6ddd66b668-q2wsn"
May 25 11:19:23.559056 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:19:23.559069    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/39cfa0b7-b3f2-4927-abb6-996baf27f85f-service-ca-bundle\") pod \"router-default-6ddd66b668-q2wsn\" (UID: \"39cfa0b7-b3f2-4927-abb6-996baf27f85f\") " pod="openshift-ingress/router-default-6ddd66b668-q2wsn"
May 25 11:19:23.559503 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:19:23.559149    2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
May 25 11:19:23.559503 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:19:23.559193    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/39cfa0b7-b3f2-4927-abb6-996baf27f85f-service-ca-bundle podName:39cfa0b7-b3f2-4927-abb6-996baf27f85f nodeName:}" failed. No retries permitted until 2026-05-25 11:19:25.559177421 +0000 UTC m=+111.076908870 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/39cfa0b7-b3f2-4927-abb6-996baf27f85f-service-ca-bundle") pod "router-default-6ddd66b668-q2wsn" (UID: "39cfa0b7-b3f2-4927-abb6-996baf27f85f") : configmap references non-existent config key: service-ca.crt
May 25 11:19:23.559503 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:19:23.559209    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/39cfa0b7-b3f2-4927-abb6-996baf27f85f-metrics-certs podName:39cfa0b7-b3f2-4927-abb6-996baf27f85f nodeName:}" failed. No retries permitted until 2026-05-25 11:19:25.559203257 +0000 UTC m=+111.076934690 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/39cfa0b7-b3f2-4927-abb6-996baf27f85f-metrics-certs") pod "router-default-6ddd66b668-q2wsn" (UID: "39cfa0b7-b3f2-4927-abb6-996baf27f85f") : secret "router-metrics-certs-default" not found
May 25 11:19:23.660338 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:19:23.660303    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6926af5f-04cd-4fed-bb10-669c46bc0c60-registry-tls\") pod \"image-registry-6f78b857d8-gcqd2\" (UID: \"6926af5f-04cd-4fed-bb10-669c46bc0c60\") " pod="openshift-image-registry/image-registry-6f78b857d8-gcqd2"
May 25 11:19:23.660469 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:19:23.660424    2576 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
May 25 11:19:23.660469 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:19:23.660439    2576 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6f78b857d8-gcqd2: secret "image-registry-tls" not found
May 25 11:19:23.660534 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:19:23.660483    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6926af5f-04cd-4fed-bb10-669c46bc0c60-registry-tls podName:6926af5f-04cd-4fed-bb10-669c46bc0c60 nodeName:}" failed. No retries permitted until 2026-05-25 11:19:25.660469328 +0000 UTC m=+111.178200762 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6926af5f-04cd-4fed-bb10-669c46bc0c60-registry-tls") pod "image-registry-6f78b857d8-gcqd2" (UID: "6926af5f-04cd-4fed-bb10-669c46bc0c60") : secret "image-registry-tls" not found
May 25 11:19:25.572188 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:19:25.572148    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/39cfa0b7-b3f2-4927-abb6-996baf27f85f-metrics-certs\") pod \"router-default-6ddd66b668-q2wsn\" (UID: \"39cfa0b7-b3f2-4927-abb6-996baf27f85f\") " pod="openshift-ingress/router-default-6ddd66b668-q2wsn"
May 25 11:19:25.572699 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:19:25.572204    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/39cfa0b7-b3f2-4927-abb6-996baf27f85f-service-ca-bundle\") pod \"router-default-6ddd66b668-q2wsn\" (UID: \"39cfa0b7-b3f2-4927-abb6-996baf27f85f\") " pod="openshift-ingress/router-default-6ddd66b668-q2wsn"
May 25 11:19:25.572699 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:19:25.572317    2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
May 25 11:19:25.572699 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:19:25.572379    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/39cfa0b7-b3f2-4927-abb6-996baf27f85f-metrics-certs podName:39cfa0b7-b3f2-4927-abb6-996baf27f85f nodeName:}" failed. No retries permitted until 2026-05-25 11:19:29.572360932 +0000 UTC m=+115.090092372 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/39cfa0b7-b3f2-4927-abb6-996baf27f85f-metrics-certs") pod "router-default-6ddd66b668-q2wsn" (UID: "39cfa0b7-b3f2-4927-abb6-996baf27f85f") : secret "router-metrics-certs-default" not found
May 25 11:19:25.572699 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:19:25.572398    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/39cfa0b7-b3f2-4927-abb6-996baf27f85f-service-ca-bundle podName:39cfa0b7-b3f2-4927-abb6-996baf27f85f nodeName:}" failed. No retries permitted until 2026-05-25 11:19:29.57238929 +0000 UTC m=+115.090120738 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/39cfa0b7-b3f2-4927-abb6-996baf27f85f-service-ca-bundle") pod "router-default-6ddd66b668-q2wsn" (UID: "39cfa0b7-b3f2-4927-abb6-996baf27f85f") : configmap references non-existent config key: service-ca.crt
May 25 11:19:25.672694 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:19:25.672671    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6926af5f-04cd-4fed-bb10-669c46bc0c60-registry-tls\") pod \"image-registry-6f78b857d8-gcqd2\" (UID: \"6926af5f-04cd-4fed-bb10-669c46bc0c60\") " pod="openshift-image-registry/image-registry-6f78b857d8-gcqd2"
May 25 11:19:25.672803 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:19:25.672780    2576 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
May 25 11:19:25.672803 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:19:25.672792    2576 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6f78b857d8-gcqd2: secret "image-registry-tls" not found
May 25 11:19:25.672868 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:19:25.672839    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6926af5f-04cd-4fed-bb10-669c46bc0c60-registry-tls podName:6926af5f-04cd-4fed-bb10-669c46bc0c60 nodeName:}" failed. No retries permitted until 2026-05-25 11:19:29.672823429 +0000 UTC m=+115.190554865 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6926af5f-04cd-4fed-bb10-669c46bc0c60-registry-tls") pod "image-registry-6f78b857d8-gcqd2" (UID: "6926af5f-04cd-4fed-bb10-669c46bc0c60") : secret "image-registry-tls" not found
May 25 11:19:29.598492 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:19:29.598460    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/39cfa0b7-b3f2-4927-abb6-996baf27f85f-service-ca-bundle\") pod \"router-default-6ddd66b668-q2wsn\" (UID: \"39cfa0b7-b3f2-4927-abb6-996baf27f85f\") " pod="openshift-ingress/router-default-6ddd66b668-q2wsn"
May 25 11:19:29.598935 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:19:29.598540    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/39cfa0b7-b3f2-4927-abb6-996baf27f85f-metrics-certs\") pod \"router-default-6ddd66b668-q2wsn\" (UID: \"39cfa0b7-b3f2-4927-abb6-996baf27f85f\") " pod="openshift-ingress/router-default-6ddd66b668-q2wsn"
May 25 11:19:29.598935 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:19:29.598614    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/39cfa0b7-b3f2-4927-abb6-996baf27f85f-service-ca-bundle podName:39cfa0b7-b3f2-4927-abb6-996baf27f85f nodeName:}" failed. No retries permitted until 2026-05-25 11:19:37.598593448 +0000 UTC m=+123.116324882 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/39cfa0b7-b3f2-4927-abb6-996baf27f85f-service-ca-bundle") pod "router-default-6ddd66b668-q2wsn" (UID: "39cfa0b7-b3f2-4927-abb6-996baf27f85f") : configmap references non-existent config key: service-ca.crt
May 25 11:19:29.598935 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:19:29.598622    2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
May 25 11:19:29.598935 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:19:29.598652    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/39cfa0b7-b3f2-4927-abb6-996baf27f85f-metrics-certs podName:39cfa0b7-b3f2-4927-abb6-996baf27f85f nodeName:}" failed. No retries permitted until 2026-05-25 11:19:37.598645418 +0000 UTC m=+123.116376852 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/39cfa0b7-b3f2-4927-abb6-996baf27f85f-metrics-certs") pod "router-default-6ddd66b668-q2wsn" (UID: "39cfa0b7-b3f2-4927-abb6-996baf27f85f") : secret "router-metrics-certs-default" not found
May 25 11:19:29.699687 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:19:29.699657    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6926af5f-04cd-4fed-bb10-669c46bc0c60-registry-tls\") pod \"image-registry-6f78b857d8-gcqd2\" (UID: \"6926af5f-04cd-4fed-bb10-669c46bc0c60\") " pod="openshift-image-registry/image-registry-6f78b857d8-gcqd2"
May 25 11:19:29.699822 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:19:29.699750    2576 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
May 25 11:19:29.699822 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:19:29.699761    2576 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6f78b857d8-gcqd2: secret "image-registry-tls" not found
May 25 11:19:29.699822 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:19:29.699815    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6926af5f-04cd-4fed-bb10-669c46bc0c60-registry-tls podName:6926af5f-04cd-4fed-bb10-669c46bc0c60 nodeName:}" failed. No retries permitted until 2026-05-25 11:19:37.699802135 +0000 UTC m=+123.217533569 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6926af5f-04cd-4fed-bb10-669c46bc0c60-registry-tls") pod "image-registry-6f78b857d8-gcqd2" (UID: "6926af5f-04cd-4fed-bb10-669c46bc0c60") : secret "image-registry-tls" not found
May 25 11:19:31.066885 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:19:31.066858    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-xn288_4ab64dbc-4ab2-4199-bf57-698ec5539507/dns-node-resolver/0.log"
May 25 11:19:31.666138 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:19:31.666110    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-nkcf5_cd7634d4-8eb9-42a5-9e01-7d94645afbd3/node-ca/0.log"
May 25 11:19:37.664246 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:19:37.664213    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/39cfa0b7-b3f2-4927-abb6-996baf27f85f-metrics-certs\") pod \"router-default-6ddd66b668-q2wsn\" (UID: \"39cfa0b7-b3f2-4927-abb6-996baf27f85f\") " pod="openshift-ingress/router-default-6ddd66b668-q2wsn"
May 25 11:19:37.664643 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:19:37.664258    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/39cfa0b7-b3f2-4927-abb6-996baf27f85f-service-ca-bundle\") pod \"router-default-6ddd66b668-q2wsn\" (UID: \"39cfa0b7-b3f2-4927-abb6-996baf27f85f\") " pod="openshift-ingress/router-default-6ddd66b668-q2wsn"
May 25 11:19:37.664643 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:19:37.664383    2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
May 25 11:19:37.664643 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:19:37.664442    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/39cfa0b7-b3f2-4927-abb6-996baf27f85f-service-ca-bundle podName:39cfa0b7-b3f2-4927-abb6-996baf27f85f nodeName:}" failed. No retries permitted until 2026-05-25 11:19:53.66442284 +0000 UTC m=+139.182154275 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/39cfa0b7-b3f2-4927-abb6-996baf27f85f-service-ca-bundle") pod "router-default-6ddd66b668-q2wsn" (UID: "39cfa0b7-b3f2-4927-abb6-996baf27f85f") : configmap references non-existent config key: service-ca.crt
May 25 11:19:37.664643 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:19:37.664456    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/39cfa0b7-b3f2-4927-abb6-996baf27f85f-metrics-certs podName:39cfa0b7-b3f2-4927-abb6-996baf27f85f nodeName:}" failed. No retries permitted until 2026-05-25 11:19:53.664450737 +0000 UTC m=+139.182182170 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/39cfa0b7-b3f2-4927-abb6-996baf27f85f-metrics-certs") pod "router-default-6ddd66b668-q2wsn" (UID: "39cfa0b7-b3f2-4927-abb6-996baf27f85f") : secret "router-metrics-certs-default" not found
May 25 11:19:37.765046 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:19:37.765016    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6926af5f-04cd-4fed-bb10-669c46bc0c60-registry-tls\") pod \"image-registry-6f78b857d8-gcqd2\" (UID: \"6926af5f-04cd-4fed-bb10-669c46bc0c60\") " pod="openshift-image-registry/image-registry-6f78b857d8-gcqd2"
May 25 11:19:37.767378 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:19:37.767351    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6926af5f-04cd-4fed-bb10-669c46bc0c60-registry-tls\") pod \"image-registry-6f78b857d8-gcqd2\" (UID: \"6926af5f-04cd-4fed-bb10-669c46bc0c60\") " pod="openshift-image-registry/image-registry-6f78b857d8-gcqd2"
May 25 11:19:37.876827 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:19:37.876803    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-srlsx\""
May 25 11:19:37.884628 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:19:37.884609    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6f78b857d8-gcqd2"
May 25 11:19:38.006750 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:19:38.006719    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6f78b857d8-gcqd2"]
May 25 11:19:38.011481 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:19:38.011449    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6926af5f_04cd_4fed_bb10_669c46bc0c60.slice/crio-7afe7219e1a04300ef4c4942d3d7cdbfed7cd9423cc1a6e478c1909ffd5ec059 WatchSource:0}: Error finding container 7afe7219e1a04300ef4c4942d3d7cdbfed7cd9423cc1a6e478c1909ffd5ec059: Status 404 returned error can't find the container with id 7afe7219e1a04300ef4c4942d3d7cdbfed7cd9423cc1a6e478c1909ffd5ec059
May 25 11:19:38.417394 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:19:38.417358    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6f78b857d8-gcqd2" event={"ID":"6926af5f-04cd-4fed-bb10-669c46bc0c60","Type":"ContainerStarted","Data":"f2266cabae3f1d3c5a3347837c6262b0b300f8ec53797ddc012f2ed4beaa35c7"}
May 25 11:19:38.417546 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:19:38.417405    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6f78b857d8-gcqd2" event={"ID":"6926af5f-04cd-4fed-bb10-669c46bc0c60","Type":"ContainerStarted","Data":"7afe7219e1a04300ef4c4942d3d7cdbfed7cd9423cc1a6e478c1909ffd5ec059"}
May 25 11:19:38.417546 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:19:38.417495    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6f78b857d8-gcqd2"
May 25 11:19:38.439220 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:19:38.439160    2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-image-registry/image-registry-6f78b857d8-gcqd2" podStartSLOduration=17.439142826 podStartE2EDuration="17.439142826s" podCreationTimestamp="2026-05-25 11:19:21 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 11:19:38.438378617 +0000 UTC m=+123.956110074" watchObservedRunningTime="2026-05-25 11:19:38.439142826 +0000 UTC m=+123.956874278"
May 25 11:19:44.817290 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:19:44.817248    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ff50c150-45a9-41c6-be52-886f2c960e85-metrics-certs\") pod \"network-metrics-daemon-5859j\" (UID: \"ff50c150-45a9-41c6-be52-886f2c960e85\") " pod="openshift-multus/network-metrics-daemon-5859j"
May 25 11:19:44.819913 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:19:44.819883    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ff50c150-45a9-41c6-be52-886f2c960e85-metrics-certs\") pod \"network-metrics-daemon-5859j\" (UID: \"ff50c150-45a9-41c6-be52-886f2c960e85\") " pod="openshift-multus/network-metrics-daemon-5859j"
May 25 11:19:44.920307 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:19:44.920270    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-bl7fp\""
May 25 11:19:44.928154 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:19:44.928139    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5859j"
May 25 11:19:45.044512 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:19:45.044481    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-5859j"]
May 25 11:19:45.047617 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:19:45.047588    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff50c150_45a9_41c6_be52_886f2c960e85.slice/crio-a2e4ef9610c3ec743e853645a2692a62b52f89d9a45e4ab799291ae721ebbd1c WatchSource:0}: Error finding container a2e4ef9610c3ec743e853645a2692a62b52f89d9a45e4ab799291ae721ebbd1c: Status 404 returned error can't find the container with id a2e4ef9610c3ec743e853645a2692a62b52f89d9a45e4ab799291ae721ebbd1c
May 25 11:19:45.434293 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:19:45.434249    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-5859j" event={"ID":"ff50c150-45a9-41c6-be52-886f2c960e85","Type":"ContainerStarted","Data":"a2e4ef9610c3ec743e853645a2692a62b52f89d9a45e4ab799291ae721ebbd1c"}
May 25 11:19:46.438474 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:19:46.438441    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-5859j" event={"ID":"ff50c150-45a9-41c6-be52-886f2c960e85","Type":"ContainerStarted","Data":"fa909d2eebbaa8f77c55a2ee98f5bf64ae94e385c3011272fe74fc026e0dd17c"}
May 25 11:19:46.438474 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:19:46.438478    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-5859j" event={"ID":"ff50c150-45a9-41c6-be52-886f2c960e85","Type":"ContainerStarted","Data":"0b7541d1ef474e4795df377aa3670e3936f409e11b0894d030bc26d350674368"}
May 25 11:19:46.453911 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:19:46.453868    2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-5859j" podStartSLOduration=130.470280637 podStartE2EDuration="2m11.453830978s" podCreationTimestamp="2026-05-25 11:17:35 +0000 UTC" firstStartedPulling="2026-05-25 11:19:45.049256415 +0000 UTC m=+130.566987854" lastFinishedPulling="2026-05-25 11:19:46.032806758 +0000 UTC m=+131.550538195" observedRunningTime="2026-05-25 11:19:46.45325236 +0000 UTC m=+131.970983817" watchObservedRunningTime="2026-05-25 11:19:46.453830978 +0000 UTC m=+131.971562433"
May 25 11:19:50.254142 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:19:50.254107    2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-7f44b6dd87-frn7j"]
May 25 11:19:50.257004 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:19:50.256989    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7f44b6dd87-frn7j"
May 25 11:19:50.259940 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:19:50.259913    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-zcv9m\""
May 25 11:19:50.272997 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:19:50.272970    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7f44b6dd87-frn7j"]
May 25 11:19:50.306755 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:19:50.306731    2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-8kl5j"]
May 25 11:19:50.309526 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:19:50.309511    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-8kl5j"
May 25 11:19:50.311566 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:19:50.311546    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
May 25 11:19:50.311678 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:19:50.311588    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-rgvrv\""
May 25 11:19:50.311854 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:19:50.311836    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
May 25 11:19:50.312018 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:19:50.312001    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
May 25 11:19:50.312067 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:19:50.312039    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
May 25 11:19:50.327731 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:19:50.327708    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-8kl5j"]
May 25 11:19:50.356079 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:19:50.356054    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7p72m\" (UniqueName: \"kubernetes.io/projected/005495d1-e386-42c5-998b-f67228636a0f-kube-api-access-7p72m\") pod \"network-check-source-7f44b6dd87-frn7j\" (UID: \"005495d1-e386-42c5-998b-f67228636a0f\") " pod="openshift-network-diagnostics/network-check-source-7f44b6dd87-frn7j"
May 25 11:19:50.367210 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:19:50.367189    2576 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6f78b857d8-gcqd2"]
May 25 11:19:50.457406 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:19:50.457384    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-7p72m\" (UniqueName: \"kubernetes.io/projected/005495d1-e386-42c5-998b-f67228636a0f-kube-api-access-7p72m\") pod \"network-check-source-7f44b6dd87-frn7j\" (UID: \"005495d1-e386-42c5-998b-f67228636a0f\") " pod="openshift-network-diagnostics/network-check-source-7f44b6dd87-frn7j"
May 25 11:19:50.457521 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:19:50.457415    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3d0e9f3f-1d89-4a43-b947-a242f4cf2e52-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8kl5j\" (UID: \"3d0e9f3f-1d89-4a43-b947-a242f4cf2e52\") " pod="openshift-insights/insights-runtime-extractor-8kl5j"
May 25 11:19:50.457521 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:19:50.457446    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3d0e9f3f-1d89-4a43-b947-a242f4cf2e52-crio-socket\") pod \"insights-runtime-extractor-8kl5j\" (UID: \"3d0e9f3f-1d89-4a43-b947-a242f4cf2e52\") " pod="openshift-insights/insights-runtime-extractor-8kl5j"
May 25 11:19:50.457521 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:19:50.457463    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3d0e9f3f-1d89-4a43-b947-a242f4cf2e52-data-volume\") pod \"insights-runtime-extractor-8kl5j\" (UID: \"3d0e9f3f-1d89-4a43-b947-a242f4cf2e52\") " pod="openshift-insights/insights-runtime-extractor-8kl5j"
May 25 11:19:50.457624 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:19:50.457547    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3d0e9f3f-1d89-4a43-b947-a242f4cf2e52-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8kl5j\" (UID: \"3d0e9f3f-1d89-4a43-b947-a242f4cf2e52\") " pod="openshift-insights/insights-runtime-extractor-8kl5j"
May 25 11:19:50.457654 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:19:50.457627    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-knzxr\" (UniqueName: \"kubernetes.io/projected/3d0e9f3f-1d89-4a43-b947-a242f4cf2e52-kube-api-access-knzxr\") pod \"insights-runtime-extractor-8kl5j\" (UID: \"3d0e9f3f-1d89-4a43-b947-a242f4cf2e52\") " pod="openshift-insights/insights-runtime-extractor-8kl5j"
May 25 11:19:50.481354 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:19:50.481329    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7p72m\" (UniqueName: \"kubernetes.io/projected/005495d1-e386-42c5-998b-f67228636a0f-kube-api-access-7p72m\") pod \"network-check-source-7f44b6dd87-frn7j\" (UID: \"005495d1-e386-42c5-998b-f67228636a0f\") " pod="openshift-network-diagnostics/network-check-source-7f44b6dd87-frn7j"
May 25 11:19:50.490683 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:19:50.490664    2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-685f589447-dtpd9"]
May 25 11:19:50.493517 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:19:50.493503    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-685f589447-dtpd9"
May 25 11:19:50.521624 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:19:50.521569    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-685f589447-dtpd9"]
May 25 11:19:50.558326 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:19:50.558301    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3d0e9f3f-1d89-4a43-b947-a242f4cf2e52-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8kl5j\" (UID: \"3d0e9f3f-1d89-4a43-b947-a242f4cf2e52\") " pod="openshift-insights/insights-runtime-extractor-8kl5j"
May 25 11:19:50.558429 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:19:50.558368    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-knzxr\" (UniqueName: \"kubernetes.io/projected/3d0e9f3f-1d89-4a43-b947-a242f4cf2e52-kube-api-access-knzxr\") pod \"insights-runtime-extractor-8kl5j\" (UID: \"3d0e9f3f-1d89-4a43-b947-a242f4cf2e52\") " pod="openshift-insights/insights-runtime-extractor-8kl5j"
May 25 11:19:50.558499 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:19:50.558425    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3d0e9f3f-1d89-4a43-b947-a242f4cf2e52-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8kl5j\" (UID: \"3d0e9f3f-1d89-4a43-b947-a242f4cf2e52\") " pod="openshift-insights/insights-runtime-extractor-8kl5j"
May 25 11:19:50.558499 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:19:50.558473    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3d0e9f3f-1d89-4a43-b947-a242f4cf2e52-crio-socket\") pod \"insights-runtime-extractor-8kl5j\" (UID: \"3d0e9f3f-1d89-4a43-b947-a242f4cf2e52\") " pod="openshift-insights/insights-runtime-extractor-8kl5j"
May 25 11:19:50.558602 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:19:50.558498    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3d0e9f3f-1d89-4a43-b947-a242f4cf2e52-data-volume\") pod \"insights-runtime-extractor-8kl5j\" (UID: \"3d0e9f3f-1d89-4a43-b947-a242f4cf2e52\") " pod="openshift-insights/insights-runtime-extractor-8kl5j"
May 25 11:19:50.558648 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:19:50.558595    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3d0e9f3f-1d89-4a43-b947-a242f4cf2e52-crio-socket\") pod \"insights-runtime-extractor-8kl5j\" (UID: \"3d0e9f3f-1d89-4a43-b947-a242f4cf2e52\") " pod="openshift-insights/insights-runtime-extractor-8kl5j"
May 25 11:19:50.558799 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:19:50.558781    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3d0e9f3f-1d89-4a43-b947-a242f4cf2e52-data-volume\") pod \"insights-runtime-extractor-8kl5j\" (UID: \"3d0e9f3f-1d89-4a43-b947-a242f4cf2e52\") " pod="openshift-insights/insights-runtime-extractor-8kl5j"
May 25 11:19:50.558859 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:19:50.558783    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3d0e9f3f-1d89-4a43-b947-a242f4cf2e52-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8kl5j\" (UID: \"3d0e9f3f-1d89-4a43-b947-a242f4cf2e52\") " pod="openshift-insights/insights-runtime-extractor-8kl5j"
May 25 11:19:50.560657 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:19:50.560639    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3d0e9f3f-1d89-4a43-b947-a242f4cf2e52-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8kl5j\" (UID: \"3d0e9f3f-1d89-4a43-b947-a242f4cf2e52\") " pod="openshift-insights/insights-runtime-extractor-8kl5j"
May 25 11:19:50.565536 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:19:50.565519    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7f44b6dd87-frn7j"
May 25 11:19:50.570985 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:19:50.570958    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-knzxr\" (UniqueName: \"kubernetes.io/projected/3d0e9f3f-1d89-4a43-b947-a242f4cf2e52-kube-api-access-knzxr\") pod \"insights-runtime-extractor-8kl5j\" (UID: \"3d0e9f3f-1d89-4a43-b947-a242f4cf2e52\") " pod="openshift-insights/insights-runtime-extractor-8kl5j"
May 25 11:19:50.617514 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:19:50.617487    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-8kl5j"
May 25 11:19:50.659429 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:19:50.659397    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e1e82e94-6cf9-4ca9-a606-fe51ed2aaf29-trusted-ca\") pod \"image-registry-685f589447-dtpd9\" (UID: \"e1e82e94-6cf9-4ca9-a606-fe51ed2aaf29\") " pod="openshift-image-registry/image-registry-685f589447-dtpd9"
May 25 11:19:50.659548 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:19:50.659446    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e1e82e94-6cf9-4ca9-a606-fe51ed2aaf29-installation-pull-secrets\") pod \"image-registry-685f589447-dtpd9\" (UID: \"e1e82e94-6cf9-4ca9-a606-fe51ed2aaf29\") " pod="openshift-image-registry/image-registry-685f589447-dtpd9"
May 25 11:19:50.659548 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:19:50.659484    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hsn7\" (UniqueName: \"kubernetes.io/projected/e1e82e94-6cf9-4ca9-a606-fe51ed2aaf29-kube-api-access-8hsn7\") pod \"image-registry-685f589447-dtpd9\" (UID: \"e1e82e94-6cf9-4ca9-a606-fe51ed2aaf29\") " pod="openshift-image-registry/image-registry-685f589447-dtpd9"
May 25 11:19:50.659548 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:19:50.659529    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e1e82e94-6cf9-4ca9-a606-fe51ed2aaf29-image-registry-private-configuration\") pod \"image-registry-685f589447-dtpd9\" (UID: \"e1e82e94-6cf9-4ca9-a606-fe51ed2aaf29\") " pod="openshift-image-registry/image-registry-685f589447-dtpd9"
May 25 11:19:50.659649 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:19:50.659619    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e1e82e94-6cf9-4ca9-a606-fe51ed2aaf29-registry-tls\") pod \"image-registry-685f589447-dtpd9\" (UID: \"e1e82e94-6cf9-4ca9-a606-fe51ed2aaf29\") " pod="openshift-image-registry/image-registry-685f589447-dtpd9"
May 25 11:19:50.659683 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:19:50.659652    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e1e82e94-6cf9-4ca9-a606-fe51ed2aaf29-registry-certificates\") pod \"image-registry-685f589447-dtpd9\" (UID: \"e1e82e94-6cf9-4ca9-a606-fe51ed2aaf29\") " pod="openshift-image-registry/image-registry-685f589447-dtpd9"
May 25 11:19:50.659683 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:19:50.659671    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e1e82e94-6cf9-4ca9-a606-fe51ed2aaf29-bound-sa-token\") pod \"image-registry-685f589447-dtpd9\" (UID: \"e1e82e94-6cf9-4ca9-a606-fe51ed2aaf29\") " pod="openshift-image-registry/image-registry-685f589447-dtpd9"
May 25 11:19:50.659745 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:19:50.659720    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e1e82e94-6cf9-4ca9-a606-fe51ed2aaf29-ca-trust-extracted\") pod \"image-registry-685f589447-dtpd9\" (UID: \"e1e82e94-6cf9-4ca9-a606-fe51ed2aaf29\") " pod="openshift-image-registry/image-registry-685f589447-dtpd9"
May 25 11:19:50.694109 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:19:50.694080    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7f44b6dd87-frn7j"]
May 25 11:19:50.696608 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:19:50.696583    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod005495d1_e386_42c5_998b_f67228636a0f.slice/crio-99455be1934226d60b8f7c2703f0e41dce385e325db5fe171e83d28c36ec95ce WatchSource:0}: Error finding container 99455be1934226d60b8f7c2703f0e41dce385e325db5fe171e83d28c36ec95ce: Status 404 returned error can't find the container with id 99455be1934226d60b8f7c2703f0e41dce385e325db5fe171e83d28c36ec95ce
May 25 11:19:50.751098 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:19:50.751075    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-8kl5j"]
May 25 11:19:50.753562 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:19:50.753541    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d0e9f3f_1d89_4a43_b947_a242f4cf2e52.slice/crio-89e2144fe9555d030606d2713d633917bb2ff939d47b0fc15e9f90fc1293cb21 WatchSource:0}: Error finding container 89e2144fe9555d030606d2713d633917bb2ff939d47b0fc15e9f90fc1293cb21: Status 404 returned error can't find the container with id 89e2144fe9555d030606d2713d633917bb2ff939d47b0fc15e9f90fc1293cb21
May 25 11:19:50.760814 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:19:50.760785    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e1e82e94-6cf9-4ca9-a606-fe51ed2aaf29-ca-trust-extracted\") pod \"image-registry-685f589447-dtpd9\" (UID: \"e1e82e94-6cf9-4ca9-a606-fe51ed2aaf29\") " pod="openshift-image-registry/image-registry-685f589447-dtpd9"
May 25 11:19:50.760897 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:19:50.760863    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e1e82e94-6cf9-4ca9-a606-fe51ed2aaf29-trusted-ca\") pod \"image-registry-685f589447-dtpd9\" (UID: \"e1e82e94-6cf9-4ca9-a606-fe51ed2aaf29\") " pod="openshift-image-registry/image-registry-685f589447-dtpd9"
May 25 11:19:50.760938 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:19:50.760892    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e1e82e94-6cf9-4ca9-a606-fe51ed2aaf29-installation-pull-secrets\") pod \"image-registry-685f589447-dtpd9\" (UID: \"e1e82e94-6cf9-4ca9-a606-fe51ed2aaf29\") " pod="openshift-image-registry/image-registry-685f589447-dtpd9"
May 25 11:19:50.760938 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:19:50.760922    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-8hsn7\" (UniqueName: \"kubernetes.io/projected/e1e82e94-6cf9-4ca9-a606-fe51ed2aaf29-kube-api-access-8hsn7\") pod \"image-registry-685f589447-dtpd9\" (UID: \"e1e82e94-6cf9-4ca9-a606-fe51ed2aaf29\") " pod="openshift-image-registry/image-registry-685f589447-dtpd9"
May 25 11:19:50.761098 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:19:50.761065    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e1e82e94-6cf9-4ca9-a606-fe51ed2aaf29-image-registry-private-configuration\") pod \"image-registry-685f589447-dtpd9\" (UID: \"e1e82e94-6cf9-4ca9-a606-fe51ed2aaf29\") " pod="openshift-image-registry/image-registry-685f589447-dtpd9"
May 25 11:19:50.761177 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:19:50.761103    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e1e82e94-6cf9-4ca9-a606-fe51ed2aaf29-ca-trust-extracted\") pod \"image-registry-685f589447-dtpd9\" (UID: \"e1e82e94-6cf9-4ca9-a606-fe51ed2aaf29\") " pod="openshift-image-registry/image-registry-685f589447-dtpd9"
May 25 11:19:50.761177 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:19:50.761140    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e1e82e94-6cf9-4ca9-a606-fe51ed2aaf29-registry-tls\") pod \"image-registry-685f589447-dtpd9\" (UID: \"e1e82e94-6cf9-4ca9-a606-fe51ed2aaf29\") " pod="openshift-image-registry/image-registry-685f589447-dtpd9"
May 25 11:19:50.761318 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:19:50.761172    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e1e82e94-6cf9-4ca9-a606-fe51ed2aaf29-registry-certificates\") pod \"image-registry-685f589447-dtpd9\" (UID: \"e1e82e94-6cf9-4ca9-a606-fe51ed2aaf29\") " pod="openshift-image-registry/image-registry-685f589447-dtpd9"
May 25 11:19:50.761318 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:19:50.761205    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e1e82e94-6cf9-4ca9-a606-fe51ed2aaf29-bound-sa-token\") pod \"image-registry-685f589447-dtpd9\" (UID: \"e1e82e94-6cf9-4ca9-a606-fe51ed2aaf29\") " pod="openshift-image-registry/image-registry-685f589447-dtpd9"
May 25 11:19:50.762058 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:19:50.762030    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e1e82e94-6cf9-4ca9-a606-fe51ed2aaf29-registry-certificates\") pod \"image-registry-685f589447-dtpd9\" (UID: \"e1e82e94-6cf9-4ca9-a606-fe51ed2aaf29\") " pod="openshift-image-registry/image-registry-685f589447-dtpd9"
May 25 11:19:50.762148 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:19:50.762125    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e1e82e94-6cf9-4ca9-a606-fe51ed2aaf29-trusted-ca\") pod \"image-registry-685f589447-dtpd9\" (UID: \"e1e82e94-6cf9-4ca9-a606-fe51ed2aaf29\") " pod="openshift-image-registry/image-registry-685f589447-dtpd9"
May 25 11:19:50.763383 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:19:50.763355    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e1e82e94-6cf9-4ca9-a606-fe51ed2aaf29-image-registry-private-configuration\") pod \"image-registry-685f589447-dtpd9\" (UID: \"e1e82e94-6cf9-4ca9-a606-fe51ed2aaf29\") " pod="openshift-image-registry/image-registry-685f589447-dtpd9"
May 25 11:19:50.763569 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:19:50.763546    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e1e82e94-6cf9-4ca9-a606-fe51ed2aaf29-installation-pull-secrets\") pod \"image-registry-685f589447-dtpd9\" (UID: \"e1e82e94-6cf9-4ca9-a606-fe51ed2aaf29\") " pod="openshift-image-registry/image-registry-685f589447-dtpd9"
May 25 11:19:50.763787 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:19:50.763769    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e1e82e94-6cf9-4ca9-a606-fe51ed2aaf29-registry-tls\") pod \"image-registry-685f589447-dtpd9\" (UID: \"e1e82e94-6cf9-4ca9-a606-fe51ed2aaf29\") " pod="openshift-image-registry/image-registry-685f589447-dtpd9"
May 25 11:19:50.773612 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:19:50.773569    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e1e82e94-6cf9-4ca9-a606-fe51ed2aaf29-bound-sa-token\") pod \"image-registry-685f589447-dtpd9\" (UID: \"e1e82e94-6cf9-4ca9-a606-fe51ed2aaf29\") " pod="openshift-image-registry/image-registry-685f589447-dtpd9"
May 25 11:19:50.780079 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:19:50.780061    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hsn7\" (UniqueName: \"kubernetes.io/projected/e1e82e94-6cf9-4ca9-a606-fe51ed2aaf29-kube-api-access-8hsn7\") pod \"image-registry-685f589447-dtpd9\" (UID: \"e1e82e94-6cf9-4ca9-a606-fe51ed2aaf29\") " pod="openshift-image-registry/image-registry-685f589447-dtpd9"
May 25 11:19:50.801783 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:19:50.801763    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-685f589447-dtpd9"
May 25 11:19:50.951491 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:19:50.951459    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-685f589447-dtpd9"]
May 25 11:19:50.954341 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:19:50.954312    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1e82e94_6cf9_4ca9_a606_fe51ed2aaf29.slice/crio-53d36014ab5369781631fde66d3b263d890a4f99238630332a9858e8738b5018 WatchSource:0}: Error finding container 53d36014ab5369781631fde66d3b263d890a4f99238630332a9858e8738b5018: Status 404 returned error can't find the container with id 53d36014ab5369781631fde66d3b263d890a4f99238630332a9858e8738b5018
May 25 11:19:51.452294 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:19:51.452241    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8kl5j" event={"ID":"3d0e9f3f-1d89-4a43-b947-a242f4cf2e52","Type":"ContainerStarted","Data":"86ed60c641d610335c3bb584575b8b35c5229ba94bf9e57c200eaff95cc39b9a"}
May 25 11:19:51.452707 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:19:51.452308    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8kl5j" event={"ID":"3d0e9f3f-1d89-4a43-b947-a242f4cf2e52","Type":"ContainerStarted","Data":"89e2144fe9555d030606d2713d633917bb2ff939d47b0fc15e9f90fc1293cb21"}
May 25 11:19:51.453514 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:19:51.453484    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-685f589447-dtpd9" event={"ID":"e1e82e94-6cf9-4ca9-a606-fe51ed2aaf29","Type":"ContainerStarted","Data":"085a4373b7eb949c620b669af82ef8594fdba41d6398d1e4248e2861fcd29b87"}
May 25 11:19:51.453514 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:19:51.453513    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-685f589447-dtpd9" event={"ID":"e1e82e94-6cf9-4ca9-a606-fe51ed2aaf29","Type":"ContainerStarted","Data":"53d36014ab5369781631fde66d3b263d890a4f99238630332a9858e8738b5018"}
May 25 11:19:51.453716 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:19:51.453578    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-685f589447-dtpd9"
May 25 11:19:51.455346 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:19:51.455269    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7f44b6dd87-frn7j" event={"ID":"005495d1-e386-42c5-998b-f67228636a0f","Type":"ContainerStarted","Data":"c222ffc89c4ccc0c7c51a422ff387271d02b5ec713c9bf40afd32fb947655361"}
May 25 11:19:51.455346 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:19:51.455333    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7f44b6dd87-frn7j" event={"ID":"005495d1-e386-42c5-998b-f67228636a0f","Type":"ContainerStarted","Data":"99455be1934226d60b8f7c2703f0e41dce385e325db5fe171e83d28c36ec95ce"}
May 25 11:19:51.474181 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:19:51.474129    2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-image-registry/image-registry-685f589447-dtpd9" podStartSLOduration=1.4741153040000001 podStartE2EDuration="1.474115304s" podCreationTimestamp="2026-05-25 11:19:50 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 11:19:51.473132129 +0000 UTC m=+136.990863585" watchObservedRunningTime="2026-05-25 11:19:51.474115304 +0000 UTC m=+136.991846761"
May 25 11:19:51.492779 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:19:51.492742    2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-7f44b6dd87-frn7j" podStartSLOduration=1.49273162 podStartE2EDuration="1.49273162s" podCreationTimestamp="2026-05-25 11:19:50 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 11:19:51.491377391 +0000 UTC m=+137.009108852" watchObservedRunningTime="2026-05-25 11:19:51.49273162 +0000 UTC m=+137.010463124"
May 25 11:19:52.459680 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:19:52.459637    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8kl5j" event={"ID":"3d0e9f3f-1d89-4a43-b947-a242f4cf2e52","Type":"ContainerStarted","Data":"fdfc74b092b07491dab6b419c4576209d0a1ecfe84ccaad187a11b7ecb9e202c"}
May 25 11:19:53.464447 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:19:53.464410    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8kl5j" event={"ID":"3d0e9f3f-1d89-4a43-b947-a242f4cf2e52","Type":"ContainerStarted","Data":"1e2e29597be5d88781714cc430a585186ebd3c564ad4d70170637fb98a1f9bc0"}
May 25 11:19:53.483248 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:19:53.483195    2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-8kl5j" podStartSLOduration=1.361951136 podStartE2EDuration="3.483180636s" podCreationTimestamp="2026-05-25 11:19:50 +0000 UTC" firstStartedPulling="2026-05-25 11:19:50.812408005 +0000 UTC m=+136.330139439" lastFinishedPulling="2026-05-25 11:19:52.933637499 +0000 UTC m=+138.451368939" observedRunningTime="2026-05-25 11:19:53.482465002 +0000 UTC m=+139.000196458" watchObservedRunningTime="2026-05-25 11:19:53.483180636 +0000 UTC m=+139.000912136"
May 25 11:19:53.686997 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:19:53.686964    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/39cfa0b7-b3f2-4927-abb6-996baf27f85f-metrics-certs\") pod \"router-default-6ddd66b668-q2wsn\" (UID: \"39cfa0b7-b3f2-4927-abb6-996baf27f85f\") " pod="openshift-ingress/router-default-6ddd66b668-q2wsn"
May 25 11:19:53.687161 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:19:53.687029    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/39cfa0b7-b3f2-4927-abb6-996baf27f85f-service-ca-bundle\") pod \"router-default-6ddd66b668-q2wsn\" (UID: \"39cfa0b7-b3f2-4927-abb6-996baf27f85f\") " pod="openshift-ingress/router-default-6ddd66b668-q2wsn"
May 25 11:19:53.687748 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:19:53.687729    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/39cfa0b7-b3f2-4927-abb6-996baf27f85f-service-ca-bundle\") pod \"router-default-6ddd66b668-q2wsn\" (UID: \"39cfa0b7-b3f2-4927-abb6-996baf27f85f\") " pod="openshift-ingress/router-default-6ddd66b668-q2wsn"
May 25 11:19:53.689421 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:19:53.689390    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/39cfa0b7-b3f2-4927-abb6-996baf27f85f-metrics-certs\") pod \"router-default-6ddd66b668-q2wsn\" (UID: \"39cfa0b7-b3f2-4927-abb6-996baf27f85f\") " pod="openshift-ingress/router-default-6ddd66b668-q2wsn"
May 25 11:19:53.981120 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:19:53.981096    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-lkc78\""
May 25 11:19:53.989431 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:19:53.989412    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-6ddd66b668-q2wsn"
May 25 11:19:54.109474 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:19:54.109421    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-6ddd66b668-q2wsn"]
May 25 11:19:54.111817 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:19:54.111789    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39cfa0b7_b3f2_4927_abb6_996baf27f85f.slice/crio-3963064619e3026059b311d7ea70c4e979b4239da0869d8d8ae95080f4d06661 WatchSource:0}: Error finding container 3963064619e3026059b311d7ea70c4e979b4239da0869d8d8ae95080f4d06661: Status 404 returned error can't find the container with id 3963064619e3026059b311d7ea70c4e979b4239da0869d8d8ae95080f4d06661
May 25 11:19:54.468643 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:19:54.468610    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-6ddd66b668-q2wsn" event={"ID":"39cfa0b7-b3f2-4927-abb6-996baf27f85f","Type":"ContainerStarted","Data":"a8de6931367a45fc7f8a134f1426b2cb74e7a3ef37e7125f32adafeb7552ce19"}
May 25 11:19:54.468643 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:19:54.468650    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-6ddd66b668-q2wsn" event={"ID":"39cfa0b7-b3f2-4927-abb6-996baf27f85f","Type":"ContainerStarted","Data":"3963064619e3026059b311d7ea70c4e979b4239da0869d8d8ae95080f4d06661"}
May 25 11:19:54.488393 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:19:54.488352    2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-ingress/router-default-6ddd66b668-q2wsn" podStartSLOduration=33.488339299 podStartE2EDuration="33.488339299s" podCreationTimestamp="2026-05-25 11:19:21 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 11:19:54.487538484 +0000 UTC m=+140.005269941" watchObservedRunningTime="2026-05-25 11:19:54.488339299 +0000 UTC m=+140.006070752"
May 25 11:19:54.990116 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:19:54.990086    2576 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-6ddd66b668-q2wsn"
May 25 11:19:54.992415 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:19:54.992395    2576 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-6ddd66b668-q2wsn"
May 25 11:19:55.471534 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:19:55.471508    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-6ddd66b668-q2wsn"
May 25 11:19:55.472751 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:19:55.472728    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-6ddd66b668-q2wsn"
May 25 11:20:00.373457 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:00.373425    2576 patch_prober.go:28] interesting pod/image-registry-6f78b857d8-gcqd2 container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
May 25 11:20:00.373817 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:00.373486    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-6f78b857d8-gcqd2" podUID="6926af5f-04cd-4fed-bb10-669c46bc0c60" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 11:20:04.435795 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:04.435762    2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-9c959559f-strz4"]
May 25 11:20:04.438683 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:04.438667    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-9c959559f-strz4"
May 25 11:20:04.441297 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:04.441253    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
May 25 11:20:04.441297 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:04.441257    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-n4vrx\""
May 25 11:20:04.441483 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:04.441257    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
May 25 11:20:04.441483 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:04.441412    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
May 25 11:20:04.442127 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:04.442104    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
May 25 11:20:04.442237 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:04.442176    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
May 25 11:20:04.454348 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:04.454329    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-9c959559f-strz4"]
May 25 11:20:04.467987 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:04.467966    2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-prk6s"]
May 25 11:20:04.469869 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:04.469855    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-prk6s"
May 25 11:20:04.473884 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:04.473863    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
May 25 11:20:04.473974 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:04.473929    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
May 25 11:20:04.474043 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:04.473864    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
May 25 11:20:04.474978 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:04.474961    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-jfmc7\""
May 25 11:20:04.558052 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:04.558028    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bd56bbf7-e0c6-420b-beea-e53649d3ff13-metrics-client-ca\") pod \"node-exporter-prk6s\" (UID: \"bd56bbf7-e0c6-420b-beea-e53649d3ff13\") " pod="openshift-monitoring/node-exporter-prk6s"
May 25 11:20:04.558154 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:04.558058    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/bd56bbf7-e0c6-420b-beea-e53649d3ff13-node-exporter-tls\") pod \"node-exporter-prk6s\" (UID: \"bd56bbf7-e0c6-420b-beea-e53649d3ff13\") " pod="openshift-monitoring/node-exporter-prk6s"
May 25 11:20:04.558154 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:04.558077    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5db1ca72-24e9-4109-a52d-3e05b9682a00-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-9c959559f-strz4\" (UID: \"5db1ca72-24e9-4109-a52d-3e05b9682a00\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-strz4"
May 25 11:20:04.558154 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:04.558133    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krnmc\" (UniqueName: \"kubernetes.io/projected/bd56bbf7-e0c6-420b-beea-e53649d3ff13-kube-api-access-krnmc\") pod \"node-exporter-prk6s\" (UID: \"bd56bbf7-e0c6-420b-beea-e53649d3ff13\") " pod="openshift-monitoring/node-exporter-prk6s"
May 25 11:20:04.558292 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:04.558156    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bd56bbf7-e0c6-420b-beea-e53649d3ff13-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-prk6s\" (UID: \"bd56bbf7-e0c6-420b-beea-e53649d3ff13\") " pod="openshift-monitoring/node-exporter-prk6s"
May 25 11:20:04.558292 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:04.558172    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/bd56bbf7-e0c6-420b-beea-e53649d3ff13-root\") pod \"node-exporter-prk6s\" (UID: \"bd56bbf7-e0c6-420b-beea-e53649d3ff13\") " pod="openshift-monitoring/node-exporter-prk6s"
May 25 11:20:04.558292 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:04.558206    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/5db1ca72-24e9-4109-a52d-3e05b9682a00-openshift-state-metrics-tls\") pod \"openshift-state-metrics-9c959559f-strz4\" (UID: \"5db1ca72-24e9-4109-a52d-3e05b9682a00\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-strz4"
May 25 11:20:04.558292 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:04.558224    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/bd56bbf7-e0c6-420b-beea-e53649d3ff13-node-exporter-wtmp\") pod \"node-exporter-prk6s\" (UID: \"bd56bbf7-e0c6-420b-beea-e53649d3ff13\") " pod="openshift-monitoring/node-exporter-prk6s"
May 25 11:20:04.558292 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:04.558239    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5db1ca72-24e9-4109-a52d-3e05b9682a00-metrics-client-ca\") pod \"openshift-state-metrics-9c959559f-strz4\" (UID: \"5db1ca72-24e9-4109-a52d-3e05b9682a00\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-strz4"
May 25 11:20:04.558292 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:04.558258    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/bd56bbf7-e0c6-420b-beea-e53649d3ff13-node-exporter-textfile\") pod \"node-exporter-prk6s\" (UID: \"bd56bbf7-e0c6-420b-beea-e53649d3ff13\") " pod="openshift-monitoring/node-exporter-prk6s"
May 25 11:20:04.558475 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:04.558296    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/bd56bbf7-e0c6-420b-beea-e53649d3ff13-node-exporter-accelerators-collector-config\") pod \"node-exporter-prk6s\" (UID: \"bd56bbf7-e0c6-420b-beea-e53649d3ff13\") " pod="openshift-monitoring/node-exporter-prk6s"
May 25 11:20:04.558475 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:04.558320    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-894g7\" (UniqueName: \"kubernetes.io/projected/5db1ca72-24e9-4109-a52d-3e05b9682a00-kube-api-access-894g7\") pod \"openshift-state-metrics-9c959559f-strz4\" (UID: \"5db1ca72-24e9-4109-a52d-3e05b9682a00\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-strz4"
May 25 11:20:04.558475 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:04.558338    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bd56bbf7-e0c6-420b-beea-e53649d3ff13-sys\") pod \"node-exporter-prk6s\" (UID: \"bd56bbf7-e0c6-420b-beea-e53649d3ff13\") " pod="openshift-monitoring/node-exporter-prk6s"
May 25 11:20:04.658742 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:04.658707    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5db1ca72-24e9-4109-a52d-3e05b9682a00-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-9c959559f-strz4\" (UID: \"5db1ca72-24e9-4109-a52d-3e05b9682a00\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-strz4"
May 25 11:20:04.658824 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:04.658761    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-krnmc\" (UniqueName: \"kubernetes.io/projected/bd56bbf7-e0c6-420b-beea-e53649d3ff13-kube-api-access-krnmc\") pod \"node-exporter-prk6s\" (UID: \"bd56bbf7-e0c6-420b-beea-e53649d3ff13\") " pod="openshift-monitoring/node-exporter-prk6s"
May 25 11:20:04.658824 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:04.658784    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bd56bbf7-e0c6-420b-beea-e53649d3ff13-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-prk6s\" (UID: \"bd56bbf7-e0c6-420b-beea-e53649d3ff13\") " pod="openshift-monitoring/node-exporter-prk6s"
May 25 11:20:04.658824 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:04.658804    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/bd56bbf7-e0c6-420b-beea-e53649d3ff13-root\") pod \"node-exporter-prk6s\" (UID: \"bd56bbf7-e0c6-420b-beea-e53649d3ff13\") " pod="openshift-monitoring/node-exporter-prk6s"
May 25 11:20:04.658824 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:04.658822    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/5db1ca72-24e9-4109-a52d-3e05b9682a00-openshift-state-metrics-tls\") pod \"openshift-state-metrics-9c959559f-strz4\" (UID: \"5db1ca72-24e9-4109-a52d-3e05b9682a00\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-strz4"
May 25 11:20:04.658998 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:04.658840    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/bd56bbf7-e0c6-420b-beea-e53649d3ff13-node-exporter-wtmp\") pod \"node-exporter-prk6s\" (UID: \"bd56bbf7-e0c6-420b-beea-e53649d3ff13\") " pod="openshift-monitoring/node-exporter-prk6s"
May 25 11:20:04.658998 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:04.658856    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5db1ca72-24e9-4109-a52d-3e05b9682a00-metrics-client-ca\") pod \"openshift-state-metrics-9c959559f-strz4\" (UID: \"5db1ca72-24e9-4109-a52d-3e05b9682a00\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-strz4"
May 25 11:20:04.658998 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:04.658876    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/bd56bbf7-e0c6-420b-beea-e53649d3ff13-node-exporter-textfile\") pod \"node-exporter-prk6s\" (UID: \"bd56bbf7-e0c6-420b-beea-e53649d3ff13\") " pod="openshift-monitoring/node-exporter-prk6s"
May 25 11:20:04.658998 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:04.658903    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/bd56bbf7-e0c6-420b-beea-e53649d3ff13-node-exporter-accelerators-collector-config\") pod \"node-exporter-prk6s\" (UID: \"bd56bbf7-e0c6-420b-beea-e53649d3ff13\") " pod="openshift-monitoring/node-exporter-prk6s"
May 25 11:20:04.658998 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:04.658914    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/bd56bbf7-e0c6-420b-beea-e53649d3ff13-root\") pod \"node-exporter-prk6s\" (UID: \"bd56bbf7-e0c6-420b-beea-e53649d3ff13\") " pod="openshift-monitoring/node-exporter-prk6s"
May 25 11:20:04.658998 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:04.658941    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-894g7\" (UniqueName: \"kubernetes.io/projected/5db1ca72-24e9-4109-a52d-3e05b9682a00-kube-api-access-894g7\") pod \"openshift-state-metrics-9c959559f-strz4\" (UID: \"5db1ca72-24e9-4109-a52d-3e05b9682a00\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-strz4"
May 25 11:20:04.658998 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:04.658968    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bd56bbf7-e0c6-420b-beea-e53649d3ff13-sys\") pod \"node-exporter-prk6s\" (UID: \"bd56bbf7-e0c6-420b-beea-e53649d3ff13\") " pod="openshift-monitoring/node-exporter-prk6s"
May 25 11:20:04.658998 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:04.658996    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bd56bbf7-e0c6-420b-beea-e53649d3ff13-metrics-client-ca\") pod \"node-exporter-prk6s\" (UID: \"bd56bbf7-e0c6-420b-beea-e53649d3ff13\") " pod="openshift-monitoring/node-exporter-prk6s"
May 25 11:20:04.659412 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:04.659027    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/bd56bbf7-e0c6-420b-beea-e53649d3ff13-node-exporter-tls\") pod \"node-exporter-prk6s\" (UID: \"bd56bbf7-e0c6-420b-beea-e53649d3ff13\") " pod="openshift-monitoring/node-exporter-prk6s"
May 25 11:20:04.659412 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:20:04.659127    2576 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
May 25 11:20:04.659412 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:20:04.659216    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bd56bbf7-e0c6-420b-beea-e53649d3ff13-node-exporter-tls podName:bd56bbf7-e0c6-420b-beea-e53649d3ff13 nodeName:}" failed. No retries permitted until 2026-05-25 11:20:05.159195711 +0000 UTC m=+150.676927152 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/bd56bbf7-e0c6-420b-beea-e53649d3ff13-node-exporter-tls") pod "node-exporter-prk6s" (UID: "bd56bbf7-e0c6-420b-beea-e53649d3ff13") : secret "node-exporter-tls" not found
May 25 11:20:04.659412 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:04.659296    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/bd56bbf7-e0c6-420b-beea-e53649d3ff13-node-exporter-textfile\") pod \"node-exporter-prk6s\" (UID: \"bd56bbf7-e0c6-420b-beea-e53649d3ff13\") " pod="openshift-monitoring/node-exporter-prk6s"
May 25 11:20:04.659610 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:04.659479    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bd56bbf7-e0c6-420b-beea-e53649d3ff13-sys\") pod \"node-exporter-prk6s\" (UID: \"bd56bbf7-e0c6-420b-beea-e53649d3ff13\") " pod="openshift-monitoring/node-exporter-prk6s"
May 25 11:20:04.659658 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:04.659612    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/bd56bbf7-e0c6-420b-beea-e53649d3ff13-node-exporter-wtmp\") pod \"node-exporter-prk6s\" (UID: \"bd56bbf7-e0c6-420b-beea-e53649d3ff13\") " pod="openshift-monitoring/node-exporter-prk6s"
May 25 11:20:04.659728 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:04.659707    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5db1ca72-24e9-4109-a52d-3e05b9682a00-metrics-client-ca\") pod \"openshift-state-metrics-9c959559f-strz4\" (UID: \"5db1ca72-24e9-4109-a52d-3e05b9682a00\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-strz4"
May 25 11:20:04.659807 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:04.659781    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/bd56bbf7-e0c6-420b-beea-e53649d3ff13-node-exporter-accelerators-collector-config\") pod \"node-exporter-prk6s\" (UID: \"bd56bbf7-e0c6-420b-beea-e53649d3ff13\") " pod="openshift-monitoring/node-exporter-prk6s"
May 25 11:20:04.659865 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:04.659833    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bd56bbf7-e0c6-420b-beea-e53649d3ff13-metrics-client-ca\") pod \"node-exporter-prk6s\" (UID: \"bd56bbf7-e0c6-420b-beea-e53649d3ff13\") " pod="openshift-monitoring/node-exporter-prk6s"
May 25 11:20:04.661354 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:04.661331    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5db1ca72-24e9-4109-a52d-3e05b9682a00-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-9c959559f-strz4\" (UID: \"5db1ca72-24e9-4109-a52d-3e05b9682a00\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-strz4"
May 25 11:20:04.661354 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:04.661341    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bd56bbf7-e0c6-420b-beea-e53649d3ff13-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-prk6s\" (UID: \"bd56bbf7-e0c6-420b-beea-e53649d3ff13\") " pod="openshift-monitoring/node-exporter-prk6s"
May 25 11:20:04.661509 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:04.661464    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/5db1ca72-24e9-4109-a52d-3e05b9682a00-openshift-state-metrics-tls\") pod \"openshift-state-metrics-9c959559f-strz4\" (UID: \"5db1ca72-24e9-4109-a52d-3e05b9682a00\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-strz4"
May 25 11:20:04.669102 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:04.669078    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-krnmc\" (UniqueName: \"kubernetes.io/projected/bd56bbf7-e0c6-420b-beea-e53649d3ff13-kube-api-access-krnmc\") pod \"node-exporter-prk6s\" (UID: \"bd56bbf7-e0c6-420b-beea-e53649d3ff13\") " pod="openshift-monitoring/node-exporter-prk6s"
May 25 11:20:04.672387 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:04.672365    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-894g7\" (UniqueName: \"kubernetes.io/projected/5db1ca72-24e9-4109-a52d-3e05b9682a00-kube-api-access-894g7\") pod \"openshift-state-metrics-9c959559f-strz4\" (UID: \"5db1ca72-24e9-4109-a52d-3e05b9682a00\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-strz4"
May 25 11:20:04.747596 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:04.747541    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-9c959559f-strz4"
May 25 11:20:04.869827 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:04.869723    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-9c959559f-strz4"]
May 25 11:20:04.874041 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:20:04.872435    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5db1ca72_24e9_4109_a52d_3e05b9682a00.slice/crio-f257131d4fabd04ceb11e9bb56a836f1281bfcdfa978c33d1485e5f5cf0f1d8d WatchSource:0}: Error finding container f257131d4fabd04ceb11e9bb56a836f1281bfcdfa978c33d1485e5f5cf0f1d8d: Status 404 returned error can't find the container with id f257131d4fabd04ceb11e9bb56a836f1281bfcdfa978c33d1485e5f5cf0f1d8d
May 25 11:20:05.163921 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:05.163892    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/bd56bbf7-e0c6-420b-beea-e53649d3ff13-node-exporter-tls\") pod \"node-exporter-prk6s\" (UID: \"bd56bbf7-e0c6-420b-beea-e53649d3ff13\") " pod="openshift-monitoring/node-exporter-prk6s"
May 25 11:20:05.166095 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:05.166066    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/bd56bbf7-e0c6-420b-beea-e53649d3ff13-node-exporter-tls\") pod \"node-exporter-prk6s\" (UID: \"bd56bbf7-e0c6-420b-beea-e53649d3ff13\") " pod="openshift-monitoring/node-exporter-prk6s"
May 25 11:20:05.378007 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:05.377977    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-prk6s"
May 25 11:20:05.386514 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:20:05.386491    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd56bbf7_e0c6_420b_beea_e53649d3ff13.slice/crio-d85d9948c840e3a46501907e62dedccfb16562bc72ad42eab87169f39d097715 WatchSource:0}: Error finding container d85d9948c840e3a46501907e62dedccfb16562bc72ad42eab87169f39d097715: Status 404 returned error can't find the container with id d85d9948c840e3a46501907e62dedccfb16562bc72ad42eab87169f39d097715
May 25 11:20:05.495893 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:05.495823    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-prk6s" event={"ID":"bd56bbf7-e0c6-420b-beea-e53649d3ff13","Type":"ContainerStarted","Data":"d85d9948c840e3a46501907e62dedccfb16562bc72ad42eab87169f39d097715"}
May 25 11:20:05.497353 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:05.497334    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-9c959559f-strz4" event={"ID":"5db1ca72-24e9-4109-a52d-3e05b9682a00","Type":"ContainerStarted","Data":"c4ed19a9989ef2577ecedf5ca17013fe659fcf59cbf77b442c3e248791251f7e"}
May 25 11:20:05.497452 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:05.497358    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-9c959559f-strz4" event={"ID":"5db1ca72-24e9-4109-a52d-3e05b9682a00","Type":"ContainerStarted","Data":"387caf4de76ea759fec6aa9fe5873df85c9f6a078cbd4458ec6e152b366c69a6"}
May 25 11:20:05.497452 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:05.497368    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-9c959559f-strz4" event={"ID":"5db1ca72-24e9-4109-a52d-3e05b9682a00","Type":"ContainerStarted","Data":"f257131d4fabd04ceb11e9bb56a836f1281bfcdfa978c33d1485e5f5cf0f1d8d"}
May 25 11:20:05.537132 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:05.537110    2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
May 25 11:20:05.540337 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:05.540322    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
May 25 11:20:05.543141 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:05.543124    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
May 25 11:20:05.543442 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:05.543426    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-m7485\""
May 25 11:20:05.544616 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:05.544595    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
May 25 11:20:05.544907 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:05.544888    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
May 25 11:20:05.544907 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:05.544901    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
May 25 11:20:05.545028 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:05.544907    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
May 25 11:20:05.545028 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:05.544890    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
May 25 11:20:05.545226 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:05.545207    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
May 25 11:20:05.545352 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:05.545211    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
May 25 11:20:05.545524 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:05.545505    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
May 25 11:20:05.557636 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:05.557614    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
May 25 11:20:05.668617 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:05.668585    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9aa35962-584b-42c7-bb93-a45d4f196267-web-config\") pod \"alertmanager-main-0\" (UID: \"9aa35962-584b-42c7-bb93-a45d4f196267\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 11:20:05.668857 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:05.668638    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9aa35962-584b-42c7-bb93-a45d4f196267-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"9aa35962-584b-42c7-bb93-a45d4f196267\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 11:20:05.668857 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:05.668676    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/9aa35962-584b-42c7-bb93-a45d4f196267-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"9aa35962-584b-42c7-bb93-a45d4f196267\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 11:20:05.668857 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:05.668775    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9aa35962-584b-42c7-bb93-a45d4f196267-tls-assets\") pod \"alertmanager-main-0\" (UID: \"9aa35962-584b-42c7-bb93-a45d4f196267\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 11:20:05.668857 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:05.668821    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/9aa35962-584b-42c7-bb93-a45d4f196267-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"9aa35962-584b-42c7-bb93-a45d4f196267\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 11:20:05.668857 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:05.668854    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9aa35962-584b-42c7-bb93-a45d4f196267-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"9aa35962-584b-42c7-bb93-a45d4f196267\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 11:20:05.669118 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:05.668903    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9aa35962-584b-42c7-bb93-a45d4f196267-config-out\") pod \"alertmanager-main-0\" (UID: \"9aa35962-584b-42c7-bb93-a45d4f196267\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 11:20:05.669118 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:05.668941    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/9aa35962-584b-42c7-bb93-a45d4f196267-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"9aa35962-584b-42c7-bb93-a45d4f196267\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 11:20:05.669118 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:05.668993    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7wh6t\" (UniqueName: \"kubernetes.io/projected/9aa35962-584b-42c7-bb93-a45d4f196267-kube-api-access-7wh6t\") pod \"alertmanager-main-0\" (UID: \"9aa35962-584b-42c7-bb93-a45d4f196267\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 11:20:05.669118 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:05.669020    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9aa35962-584b-42c7-bb93-a45d4f196267-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"9aa35962-584b-42c7-bb93-a45d4f196267\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 11:20:05.669118 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:05.669044    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/9aa35962-584b-42c7-bb93-a45d4f196267-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"9aa35962-584b-42c7-bb93-a45d4f196267\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 11:20:05.669118 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:05.669076    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/9aa35962-584b-42c7-bb93-a45d4f196267-config-volume\") pod \"alertmanager-main-0\" (UID: \"9aa35962-584b-42c7-bb93-a45d4f196267\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 11:20:05.669453 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:05.669150    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9aa35962-584b-42c7-bb93-a45d4f196267-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"9aa35962-584b-42c7-bb93-a45d4f196267\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 11:20:05.769850 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:05.769790    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9aa35962-584b-42c7-bb93-a45d4f196267-config-out\") pod \"alertmanager-main-0\" (UID: \"9aa35962-584b-42c7-bb93-a45d4f196267\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 11:20:05.769850 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:05.769820    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/9aa35962-584b-42c7-bb93-a45d4f196267-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"9aa35962-584b-42c7-bb93-a45d4f196267\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 11:20:05.770048 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:05.769863    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-7wh6t\" (UniqueName: \"kubernetes.io/projected/9aa35962-584b-42c7-bb93-a45d4f196267-kube-api-access-7wh6t\") pod \"alertmanager-main-0\" (UID: \"9aa35962-584b-42c7-bb93-a45d4f196267\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 11:20:05.770048 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:05.769889    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9aa35962-584b-42c7-bb93-a45d4f196267-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"9aa35962-584b-42c7-bb93-a45d4f196267\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 11:20:05.770166 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:05.770046    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/9aa35962-584b-42c7-bb93-a45d4f196267-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"9aa35962-584b-42c7-bb93-a45d4f196267\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 11:20:05.770166 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:05.770092    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/9aa35962-584b-42c7-bb93-a45d4f196267-config-volume\") pod \"alertmanager-main-0\" (UID: \"9aa35962-584b-42c7-bb93-a45d4f196267\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 11:20:05.770166 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:05.770127    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9aa35962-584b-42c7-bb93-a45d4f196267-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"9aa35962-584b-42c7-bb93-a45d4f196267\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 11:20:05.770166 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:05.770154    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9aa35962-584b-42c7-bb93-a45d4f196267-web-config\") pod \"alertmanager-main-0\" (UID: \"9aa35962-584b-42c7-bb93-a45d4f196267\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 11:20:05.770406 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:05.770178    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9aa35962-584b-42c7-bb93-a45d4f196267-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"9aa35962-584b-42c7-bb93-a45d4f196267\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 11:20:05.770406 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:05.770214    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/9aa35962-584b-42c7-bb93-a45d4f196267-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"9aa35962-584b-42c7-bb93-a45d4f196267\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 11:20:05.770406 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:05.770292    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9aa35962-584b-42c7-bb93-a45d4f196267-tls-assets\") pod \"alertmanager-main-0\" (UID: \"9aa35962-584b-42c7-bb93-a45d4f196267\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 11:20:05.770406 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:05.770326    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/9aa35962-584b-42c7-bb93-a45d4f196267-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"9aa35962-584b-42c7-bb93-a45d4f196267\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 11:20:05.770406 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:05.770358    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9aa35962-584b-42c7-bb93-a45d4f196267-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"9aa35962-584b-42c7-bb93-a45d4f196267\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 11:20:05.770650 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:05.770406    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/9aa35962-584b-42c7-bb93-a45d4f196267-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"9aa35962-584b-42c7-bb93-a45d4f196267\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 11:20:05.770650 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:20:05.770508    2576 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found
May 25 11:20:05.770650 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:20:05.770564    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9aa35962-584b-42c7-bb93-a45d4f196267-secret-alertmanager-main-tls podName:9aa35962-584b-42c7-bb93-a45d4f196267 nodeName:}" failed. No retries permitted until 2026-05-25 11:20:06.270545411 +0000 UTC m=+151.788276852 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/9aa35962-584b-42c7-bb93-a45d4f196267-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "9aa35962-584b-42c7-bb93-a45d4f196267") : secret "alertmanager-main-tls" not found
May 25 11:20:05.770861 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:05.770841    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9aa35962-584b-42c7-bb93-a45d4f196267-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"9aa35962-584b-42c7-bb93-a45d4f196267\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 11:20:05.771191 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:20:05.771020    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9aa35962-584b-42c7-bb93-a45d4f196267-alertmanager-trusted-ca-bundle podName:9aa35962-584b-42c7-bb93-a45d4f196267 nodeName:}" failed. No retries permitted until 2026-05-25 11:20:06.27100101 +0000 UTC m=+151.788732462 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "alertmanager-trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/9aa35962-584b-42c7-bb93-a45d4f196267-alertmanager-trusted-ca-bundle") pod "alertmanager-main-0" (UID: "9aa35962-584b-42c7-bb93-a45d4f196267") : configmap references non-existent config key: ca-bundle.crt
May 25 11:20:05.773184 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:05.773099    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9aa35962-584b-42c7-bb93-a45d4f196267-config-out\") pod \"alertmanager-main-0\" (UID: \"9aa35962-584b-42c7-bb93-a45d4f196267\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 11:20:05.773184 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:05.773132    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/9aa35962-584b-42c7-bb93-a45d4f196267-config-volume\") pod \"alertmanager-main-0\" (UID: \"9aa35962-584b-42c7-bb93-a45d4f196267\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 11:20:05.773388 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:05.773368    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9aa35962-584b-42c7-bb93-a45d4f196267-tls-assets\") pod \"alertmanager-main-0\" (UID: \"9aa35962-584b-42c7-bb93-a45d4f196267\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 11:20:05.773431 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:05.773415    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9aa35962-584b-42c7-bb93-a45d4f196267-web-config\") pod \"alertmanager-main-0\" (UID: \"9aa35962-584b-42c7-bb93-a45d4f196267\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 11:20:05.773726 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:05.773701    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9aa35962-584b-42c7-bb93-a45d4f196267-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"9aa35962-584b-42c7-bb93-a45d4f196267\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 11:20:05.773878 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:05.773856    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9aa35962-584b-42c7-bb93-a45d4f196267-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"9aa35962-584b-42c7-bb93-a45d4f196267\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 11:20:05.774163 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:05.774141    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/9aa35962-584b-42c7-bb93-a45d4f196267-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"9aa35962-584b-42c7-bb93-a45d4f196267\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 11:20:05.775060 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:05.775039    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/9aa35962-584b-42c7-bb93-a45d4f196267-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"9aa35962-584b-42c7-bb93-a45d4f196267\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 11:20:05.779767 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:05.779750    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wh6t\" (UniqueName: \"kubernetes.io/projected/9aa35962-584b-42c7-bb93-a45d4f196267-kube-api-access-7wh6t\") pod \"alertmanager-main-0\" (UID: \"9aa35962-584b-42c7-bb93-a45d4f196267\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 11:20:06.274403 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:06.274377    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9aa35962-584b-42c7-bb93-a45d4f196267-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"9aa35962-584b-42c7-bb93-a45d4f196267\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 11:20:06.274557 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:06.274423    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/9aa35962-584b-42c7-bb93-a45d4f196267-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"9aa35962-584b-42c7-bb93-a45d4f196267\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 11:20:06.275474 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:06.275450    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9aa35962-584b-42c7-bb93-a45d4f196267-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"9aa35962-584b-42c7-bb93-a45d4f196267\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 11:20:06.277174 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:06.277154    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/9aa35962-584b-42c7-bb93-a45d4f196267-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"9aa35962-584b-42c7-bb93-a45d4f196267\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 11:20:06.448652 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:06.448626    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
May 25 11:20:06.503453 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:06.503418    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-9c959559f-strz4" event={"ID":"5db1ca72-24e9-4109-a52d-3e05b9682a00","Type":"ContainerStarted","Data":"b258824a4f7d268e621990771fd272da8aabc94be862d214bd0b358303d5b0b0"}
May 25 11:20:06.504877 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:06.504852    2576 generic.go:358] "Generic (PLEG): container finished" podID="bd56bbf7-e0c6-420b-beea-e53649d3ff13" containerID="10d3ce28fe4b87f24f19b0af3ab852661a63487afe5937c08096bddd35435131" exitCode=0
May 25 11:20:06.504976 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:06.504890    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-prk6s" event={"ID":"bd56bbf7-e0c6-420b-beea-e53649d3ff13","Type":"ContainerDied","Data":"10d3ce28fe4b87f24f19b0af3ab852661a63487afe5937c08096bddd35435131"}
May 25 11:20:06.536444 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:06.536392    2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-9c959559f-strz4" podStartSLOduration=1.26303975 podStartE2EDuration="2.536373501s" podCreationTimestamp="2026-05-25 11:20:04 +0000 UTC" firstStartedPulling="2026-05-25 11:20:04.969156581 +0000 UTC m=+150.486888015" lastFinishedPulling="2026-05-25 11:20:06.242490329 +0000 UTC m=+151.760221766" observedRunningTime="2026-05-25 11:20:06.530346768 +0000 UTC m=+152.048078249" watchObservedRunningTime="2026-05-25 11:20:06.536373501 +0000 UTC m=+152.054104958"
May 25 11:20:06.587007 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:06.586952    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
May 25 11:20:06.589991 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:20:06.589965    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9aa35962_584b_42c7_bb93_a45d4f196267.slice/crio-80dbc3b16a4378a3f980291bc8a723b8e83723d6f470b700f70e6b308920fe3b WatchSource:0}: Error finding container 80dbc3b16a4378a3f980291bc8a723b8e83723d6f470b700f70e6b308920fe3b: Status 404 returned error can't find the container with id 80dbc3b16a4378a3f980291bc8a723b8e83723d6f470b700f70e6b308920fe3b
May 25 11:20:07.515920 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:07.515880    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-prk6s" event={"ID":"bd56bbf7-e0c6-420b-beea-e53649d3ff13","Type":"ContainerStarted","Data":"62ac06e4ebe7836f07156504e5996d11644f7e7be0b37320c26bef064b485ac3"}
May 25 11:20:07.515920 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:07.515922    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-prk6s" event={"ID":"bd56bbf7-e0c6-420b-beea-e53649d3ff13","Type":"ContainerStarted","Data":"1f91cee930199ee88a3fa5b4401e40a75c1f6fd135d9e8043b1a168c4e0e7f43"}
May 25 11:20:07.517101 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:07.517074    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9aa35962-584b-42c7-bb93-a45d4f196267","Type":"ContainerStarted","Data":"80dbc3b16a4378a3f980291bc8a723b8e83723d6f470b700f70e6b308920fe3b"}
May 25 11:20:07.540058 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:07.540031    2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-65f4778c88-nlwjj"]
May 25 11:20:07.543601 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:07.543552    2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-prk6s" podStartSLOduration=2.666782735 podStartE2EDuration="3.543538024s" podCreationTimestamp="2026-05-25 11:20:04 +0000 UTC" firstStartedPulling="2026-05-25 11:20:05.3886726 +0000 UTC m=+150.906404037" lastFinishedPulling="2026-05-25 11:20:06.265427889 +0000 UTC m=+151.783159326" observedRunningTime="2026-05-25 11:20:07.541618758 +0000 UTC m=+153.059350215" watchObservedRunningTime="2026-05-25 11:20:07.543538024 +0000 UTC m=+153.061269483"
May 25 11:20:07.544892 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:07.544875    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-65f4778c88-nlwjj"
May 25 11:20:07.547176 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:07.547155    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
May 25 11:20:07.547319 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:07.547221    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
May 25 11:20:07.547319 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:07.547227    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-pq2p6\""
May 25 11:20:07.547319 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:07.547236    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
May 25 11:20:07.547480 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:07.547338    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
May 25 11:20:07.547480 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:07.547419    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-3h9a6am3cg6a4\""
May 25 11:20:07.547480 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:07.547429    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
May 25 11:20:07.556574 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:07.556552    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-65f4778c88-nlwjj"]
May 25 11:20:07.687189 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:07.687156    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6c5db401-082d-48c0-8393-56392d8afbcd-secret-grpc-tls\") pod \"thanos-querier-65f4778c88-nlwjj\" (UID: \"6c5db401-082d-48c0-8393-56392d8afbcd\") " pod="openshift-monitoring/thanos-querier-65f4778c88-nlwjj"
May 25 11:20:07.687189 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:07.687189    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/6c5db401-082d-48c0-8393-56392d8afbcd-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-65f4778c88-nlwjj\" (UID: \"6c5db401-082d-48c0-8393-56392d8afbcd\") " pod="openshift-monitoring/thanos-querier-65f4778c88-nlwjj"
May 25 11:20:07.687414 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:07.687209    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4lsng\" (UniqueName: \"kubernetes.io/projected/6c5db401-082d-48c0-8393-56392d8afbcd-kube-api-access-4lsng\") pod \"thanos-querier-65f4778c88-nlwjj\" (UID: \"6c5db401-082d-48c0-8393-56392d8afbcd\") " pod="openshift-monitoring/thanos-querier-65f4778c88-nlwjj"
May 25 11:20:07.687414 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:07.687346    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6c5db401-082d-48c0-8393-56392d8afbcd-metrics-client-ca\") pod \"thanos-querier-65f4778c88-nlwjj\" (UID: \"6c5db401-082d-48c0-8393-56392d8afbcd\") " pod="openshift-monitoring/thanos-querier-65f4778c88-nlwjj"
May 25 11:20:07.687414 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:07.687371    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/6c5db401-082d-48c0-8393-56392d8afbcd-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-65f4778c88-nlwjj\" (UID: \"6c5db401-082d-48c0-8393-56392d8afbcd\") " pod="openshift-monitoring/thanos-querier-65f4778c88-nlwjj"
May 25 11:20:07.687569 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:07.687506    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/6c5db401-082d-48c0-8393-56392d8afbcd-secret-thanos-querier-tls\") pod \"thanos-querier-65f4778c88-nlwjj\" (UID: \"6c5db401-082d-48c0-8393-56392d8afbcd\") " pod="openshift-monitoring/thanos-querier-65f4778c88-nlwjj"
May 25 11:20:07.687569 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:07.687553    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6c5db401-082d-48c0-8393-56392d8afbcd-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-65f4778c88-nlwjj\" (UID: \"6c5db401-082d-48c0-8393-56392d8afbcd\") " pod="openshift-monitoring/thanos-querier-65f4778c88-nlwjj"
May 25 11:20:07.687668 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:07.687616    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6c5db401-082d-48c0-8393-56392d8afbcd-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-65f4778c88-nlwjj\" (UID: \"6c5db401-082d-48c0-8393-56392d8afbcd\") " pod="openshift-monitoring/thanos-querier-65f4778c88-nlwjj"
May 25 11:20:07.788236 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:07.788214    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-4lsng\" (UniqueName: \"kubernetes.io/projected/6c5db401-082d-48c0-8393-56392d8afbcd-kube-api-access-4lsng\") pod \"thanos-querier-65f4778c88-nlwjj\" (UID: \"6c5db401-082d-48c0-8393-56392d8afbcd\") " pod="openshift-monitoring/thanos-querier-65f4778c88-nlwjj"
May 25 11:20:07.788351 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:07.788247    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6c5db401-082d-48c0-8393-56392d8afbcd-metrics-client-ca\") pod \"thanos-querier-65f4778c88-nlwjj\" (UID: \"6c5db401-082d-48c0-8393-56392d8afbcd\") " pod="openshift-monitoring/thanos-querier-65f4778c88-nlwjj"
May 25 11:20:07.788351 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:07.788268    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/6c5db401-082d-48c0-8393-56392d8afbcd-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-65f4778c88-nlwjj\" (UID: \"6c5db401-082d-48c0-8393-56392d8afbcd\") " pod="openshift-monitoring/thanos-querier-65f4778c88-nlwjj"
May 25 11:20:07.788462 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:07.788380    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/6c5db401-082d-48c0-8393-56392d8afbcd-secret-thanos-querier-tls\") pod \"thanos-querier-65f4778c88-nlwjj\" (UID: \"6c5db401-082d-48c0-8393-56392d8afbcd\") " pod="openshift-monitoring/thanos-querier-65f4778c88-nlwjj"
May 25 11:20:07.788462 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:07.788415    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6c5db401-082d-48c0-8393-56392d8afbcd-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-65f4778c88-nlwjj\" (UID: \"6c5db401-082d-48c0-8393-56392d8afbcd\") " pod="openshift-monitoring/thanos-querier-65f4778c88-nlwjj"
May 25 11:20:07.788462 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:07.788439    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6c5db401-082d-48c0-8393-56392d8afbcd-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-65f4778c88-nlwjj\" (UID: \"6c5db401-082d-48c0-8393-56392d8afbcd\") " pod="openshift-monitoring/thanos-querier-65f4778c88-nlwjj"
May 25 11:20:07.788619 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:07.788495    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6c5db401-082d-48c0-8393-56392d8afbcd-secret-grpc-tls\") pod \"thanos-querier-65f4778c88-nlwjj\" (UID: \"6c5db401-082d-48c0-8393-56392d8afbcd\") " pod="openshift-monitoring/thanos-querier-65f4778c88-nlwjj"
May 25 11:20:07.788619 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:07.788522    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/6c5db401-082d-48c0-8393-56392d8afbcd-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-65f4778c88-nlwjj\" (UID: \"6c5db401-082d-48c0-8393-56392d8afbcd\") " pod="openshift-monitoring/thanos-querier-65f4778c88-nlwjj"
May 25 11:20:07.789044 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:07.789020    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6c5db401-082d-48c0-8393-56392d8afbcd-metrics-client-ca\") pod \"thanos-querier-65f4778c88-nlwjj\" (UID: \"6c5db401-082d-48c0-8393-56392d8afbcd\") " pod="openshift-monitoring/thanos-querier-65f4778c88-nlwjj"
May 25 11:20:07.790537 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:07.790512    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/6c5db401-082d-48c0-8393-56392d8afbcd-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-65f4778c88-nlwjj\" (UID: \"6c5db401-082d-48c0-8393-56392d8afbcd\") " pod="openshift-monitoring/thanos-querier-65f4778c88-nlwjj"
May 25 11:20:07.790721 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:07.790702    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/6c5db401-082d-48c0-8393-56392d8afbcd-secret-thanos-querier-tls\") pod \"thanos-querier-65f4778c88-nlwjj\" (UID: \"6c5db401-082d-48c0-8393-56392d8afbcd\") " pod="openshift-monitoring/thanos-querier-65f4778c88-nlwjj"
May 25 11:20:07.791027 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:07.791010    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6c5db401-082d-48c0-8393-56392d8afbcd-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-65f4778c88-nlwjj\" (UID: \"6c5db401-082d-48c0-8393-56392d8afbcd\") " pod="openshift-monitoring/thanos-querier-65f4778c88-nlwjj"
May 25 11:20:07.791088 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:07.791054    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/6c5db401-082d-48c0-8393-56392d8afbcd-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-65f4778c88-nlwjj\" (UID: \"6c5db401-082d-48c0-8393-56392d8afbcd\") " pod="openshift-monitoring/thanos-querier-65f4778c88-nlwjj"
May 25 11:20:07.791245 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:07.791225    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6c5db401-082d-48c0-8393-56392d8afbcd-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-65f4778c88-nlwjj\" (UID: \"6c5db401-082d-48c0-8393-56392d8afbcd\") " pod="openshift-monitoring/thanos-querier-65f4778c88-nlwjj"
May 25 11:20:07.791336 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:07.791321    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6c5db401-082d-48c0-8393-56392d8afbcd-secret-grpc-tls\") pod \"thanos-querier-65f4778c88-nlwjj\" (UID: \"6c5db401-082d-48c0-8393-56392d8afbcd\") " pod="openshift-monitoring/thanos-querier-65f4778c88-nlwjj"
May 25 11:20:07.797894 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:07.797876    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4lsng\" (UniqueName: \"kubernetes.io/projected/6c5db401-082d-48c0-8393-56392d8afbcd-kube-api-access-4lsng\") pod \"thanos-querier-65f4778c88-nlwjj\" (UID: \"6c5db401-082d-48c0-8393-56392d8afbcd\") " pod="openshift-monitoring/thanos-querier-65f4778c88-nlwjj"
May 25 11:20:07.855792 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:07.855772    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-65f4778c88-nlwjj"
May 25 11:20:07.977748 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:07.977709    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-65f4778c88-nlwjj"]
May 25 11:20:07.980271 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:20:07.980243    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c5db401_082d_48c0_8393_56392d8afbcd.slice/crio-642bb1e7180270c46fa489fb739e44ac821988d3c8e746aa01bc3754b84a9d13 WatchSource:0}: Error finding container 642bb1e7180270c46fa489fb739e44ac821988d3c8e746aa01bc3754b84a9d13: Status 404 returned error can't find the container with id 642bb1e7180270c46fa489fb739e44ac821988d3c8e746aa01bc3754b84a9d13
May 25 11:20:08.521481 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:08.521450    2576 generic.go:358] "Generic (PLEG): container finished" podID="9aa35962-584b-42c7-bb93-a45d4f196267" containerID="11195249a844e7bf61ca71b8650d121ef9bec7e1c30562091b6189bb0998d647" exitCode=0
May 25 11:20:08.521910 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:08.521530    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9aa35962-584b-42c7-bb93-a45d4f196267","Type":"ContainerDied","Data":"11195249a844e7bf61ca71b8650d121ef9bec7e1c30562091b6189bb0998d647"}
May 25 11:20:08.522798 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:08.522758    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-65f4778c88-nlwjj" event={"ID":"6c5db401-082d-48c0-8393-56392d8afbcd","Type":"ContainerStarted","Data":"642bb1e7180270c46fa489fb739e44ac821988d3c8e746aa01bc3754b84a9d13"}
May 25 11:20:08.879925 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:08.879856    2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-84cc7cb899-fs5c9"]
May 25 11:20:08.883016 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:08.882993    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-84cc7cb899-fs5c9"
May 25 11:20:08.892443 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:08.892421    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-nzb66\""
May 25 11:20:08.892555 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:08.892460    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
May 25 11:20:08.892621 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:08.892556    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
May 25 11:20:08.892714 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:08.892696    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
May 25 11:20:08.893380 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:08.893363    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
May 25 11:20:08.894527 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:08.894510    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-bdk0trrirqfhc\""
May 25 11:20:08.901373 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:08.901348    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-84cc7cb899-fs5c9"]
May 25 11:20:08.997699 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:08.997677    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/c39eb3b0-777d-4f3c-b80f-a0b1607419d9-audit-log\") pod \"metrics-server-84cc7cb899-fs5c9\" (UID: \"c39eb3b0-777d-4f3c-b80f-a0b1607419d9\") " pod="openshift-monitoring/metrics-server-84cc7cb899-fs5c9"
May 25 11:20:08.997822 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:08.997707    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/c39eb3b0-777d-4f3c-b80f-a0b1607419d9-secret-metrics-server-tls\") pod \"metrics-server-84cc7cb899-fs5c9\" (UID: \"c39eb3b0-777d-4f3c-b80f-a0b1607419d9\") " pod="openshift-monitoring/metrics-server-84cc7cb899-fs5c9"
May 25 11:20:08.997822 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:08.997734    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/c39eb3b0-777d-4f3c-b80f-a0b1607419d9-secret-metrics-server-client-certs\") pod \"metrics-server-84cc7cb899-fs5c9\" (UID: \"c39eb3b0-777d-4f3c-b80f-a0b1607419d9\") " pod="openshift-monitoring/metrics-server-84cc7cb899-fs5c9"
May 25 11:20:08.997822 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:08.997778    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/c39eb3b0-777d-4f3c-b80f-a0b1607419d9-client-ca-bundle\") pod \"metrics-server-84cc7cb899-fs5c9\" (UID: \"c39eb3b0-777d-4f3c-b80f-a0b1607419d9\") " pod="openshift-monitoring/metrics-server-84cc7cb899-fs5c9"
May 25 11:20:08.997954 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:08.997823    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/c39eb3b0-777d-4f3c-b80f-a0b1607419d9-metrics-server-audit-profiles\") pod \"metrics-server-84cc7cb899-fs5c9\" (UID: \"c39eb3b0-777d-4f3c-b80f-a0b1607419d9\") " pod="openshift-monitoring/metrics-server-84cc7cb899-fs5c9"
May 25 11:20:08.997954 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:08.997857    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmbs6\" (UniqueName: \"kubernetes.io/projected/c39eb3b0-777d-4f3c-b80f-a0b1607419d9-kube-api-access-rmbs6\") pod \"metrics-server-84cc7cb899-fs5c9\" (UID: \"c39eb3b0-777d-4f3c-b80f-a0b1607419d9\") " pod="openshift-monitoring/metrics-server-84cc7cb899-fs5c9"
May 25 11:20:08.997954 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:08.997887    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c39eb3b0-777d-4f3c-b80f-a0b1607419d9-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-84cc7cb899-fs5c9\" (UID: \"c39eb3b0-777d-4f3c-b80f-a0b1607419d9\") " pod="openshift-monitoring/metrics-server-84cc7cb899-fs5c9"
May 25 11:20:09.098535 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:09.098506    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/c39eb3b0-777d-4f3c-b80f-a0b1607419d9-audit-log\") pod \"metrics-server-84cc7cb899-fs5c9\" (UID: \"c39eb3b0-777d-4f3c-b80f-a0b1607419d9\") " pod="openshift-monitoring/metrics-server-84cc7cb899-fs5c9"
May 25 11:20:09.098666 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:09.098553    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/c39eb3b0-777d-4f3c-b80f-a0b1607419d9-secret-metrics-server-tls\") pod \"metrics-server-84cc7cb899-fs5c9\" (UID: \"c39eb3b0-777d-4f3c-b80f-a0b1607419d9\") " pod="openshift-monitoring/metrics-server-84cc7cb899-fs5c9"
May 25 11:20:09.098666 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:09.098590    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/c39eb3b0-777d-4f3c-b80f-a0b1607419d9-secret-metrics-server-client-certs\") pod \"metrics-server-84cc7cb899-fs5c9\" (UID: \"c39eb3b0-777d-4f3c-b80f-a0b1607419d9\") " pod="openshift-monitoring/metrics-server-84cc7cb899-fs5c9"
May 25 11:20:09.098666 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:09.098622    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/c39eb3b0-777d-4f3c-b80f-a0b1607419d9-client-ca-bundle\") pod \"metrics-server-84cc7cb899-fs5c9\" (UID: \"c39eb3b0-777d-4f3c-b80f-a0b1607419d9\") " pod="openshift-monitoring/metrics-server-84cc7cb899-fs5c9"
May 25 11:20:09.098844 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:09.098803    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/c39eb3b0-777d-4f3c-b80f-a0b1607419d9-metrics-server-audit-profiles\") pod \"metrics-server-84cc7cb899-fs5c9\" (UID: \"c39eb3b0-777d-4f3c-b80f-a0b1607419d9\") " pod="openshift-monitoring/metrics-server-84cc7cb899-fs5c9"
May 25 11:20:09.098899 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:09.098860    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-rmbs6\" (UniqueName: \"kubernetes.io/projected/c39eb3b0-777d-4f3c-b80f-a0b1607419d9-kube-api-access-rmbs6\") pod \"metrics-server-84cc7cb899-fs5c9\" (UID: \"c39eb3b0-777d-4f3c-b80f-a0b1607419d9\") " pod="openshift-monitoring/metrics-server-84cc7cb899-fs5c9"
May 25 11:20:09.098955 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:09.098909    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c39eb3b0-777d-4f3c-b80f-a0b1607419d9-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-84cc7cb899-fs5c9\" (UID: \"c39eb3b0-777d-4f3c-b80f-a0b1607419d9\") " pod="openshift-monitoring/metrics-server-84cc7cb899-fs5c9"
May 25 11:20:09.098955 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:09.098923    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/c39eb3b0-777d-4f3c-b80f-a0b1607419d9-audit-log\") pod \"metrics-server-84cc7cb899-fs5c9\" (UID: \"c39eb3b0-777d-4f3c-b80f-a0b1607419d9\") " pod="openshift-monitoring/metrics-server-84cc7cb899-fs5c9"
May 25 11:20:09.099739 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:09.099715    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c39eb3b0-777d-4f3c-b80f-a0b1607419d9-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-84cc7cb899-fs5c9\" (UID: \"c39eb3b0-777d-4f3c-b80f-a0b1607419d9\") " pod="openshift-monitoring/metrics-server-84cc7cb899-fs5c9"
May 25 11:20:09.099839 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:09.099795    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/c39eb3b0-777d-4f3c-b80f-a0b1607419d9-metrics-server-audit-profiles\") pod \"metrics-server-84cc7cb899-fs5c9\" (UID: \"c39eb3b0-777d-4f3c-b80f-a0b1607419d9\") " pod="openshift-monitoring/metrics-server-84cc7cb899-fs5c9"
May 25 11:20:09.101462 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:09.101438    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/c39eb3b0-777d-4f3c-b80f-a0b1607419d9-secret-metrics-server-client-certs\") pod \"metrics-server-84cc7cb899-fs5c9\" (UID: \"c39eb3b0-777d-4f3c-b80f-a0b1607419d9\") " pod="openshift-monitoring/metrics-server-84cc7cb899-fs5c9"
May 25 11:20:09.101574 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:09.101519    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/c39eb3b0-777d-4f3c-b80f-a0b1607419d9-secret-metrics-server-tls\") pod \"metrics-server-84cc7cb899-fs5c9\" (UID: \"c39eb3b0-777d-4f3c-b80f-a0b1607419d9\") " pod="openshift-monitoring/metrics-server-84cc7cb899-fs5c9"
May 25 11:20:09.101874 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:09.101854    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/c39eb3b0-777d-4f3c-b80f-a0b1607419d9-client-ca-bundle\") pod \"metrics-server-84cc7cb899-fs5c9\" (UID: \"c39eb3b0-777d-4f3c-b80f-a0b1607419d9\") " pod="openshift-monitoring/metrics-server-84cc7cb899-fs5c9"
May 25 11:20:09.110202 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:09.110182    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmbs6\" (UniqueName: \"kubernetes.io/projected/c39eb3b0-777d-4f3c-b80f-a0b1607419d9-kube-api-access-rmbs6\") pod \"metrics-server-84cc7cb899-fs5c9\" (UID: \"c39eb3b0-777d-4f3c-b80f-a0b1607419d9\") " pod="openshift-monitoring/metrics-server-84cc7cb899-fs5c9"
May 25 11:20:09.194180 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:09.194122    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-84cc7cb899-fs5c9"
May 25 11:20:09.224696 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:09.224600    2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-665c9c8584-mjncd"]
May 25 11:20:09.228964 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:09.228870    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-665c9c8584-mjncd"
May 25 11:20:09.231564 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:09.231517    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
May 25 11:20:09.231564 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:09.231517    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"default-dockercfg-6n8bn\""
May 25 11:20:09.235324 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:09.235252    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-665c9c8584-mjncd"]
May 25 11:20:09.354486 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:09.354445    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-84cc7cb899-fs5c9"]
May 25 11:20:09.401845 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:09.401701    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c3f2ca6f-4b0a-47ae-ae8f-c3fce46b3e8c-monitoring-plugin-cert\") pod \"monitoring-plugin-665c9c8584-mjncd\" (UID: \"c3f2ca6f-4b0a-47ae-ae8f-c3fce46b3e8c\") " pod="openshift-monitoring/monitoring-plugin-665c9c8584-mjncd"
May 25 11:20:09.503079 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:09.503007    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c3f2ca6f-4b0a-47ae-ae8f-c3fce46b3e8c-monitoring-plugin-cert\") pod \"monitoring-plugin-665c9c8584-mjncd\" (UID: \"c3f2ca6f-4b0a-47ae-ae8f-c3fce46b3e8c\") " pod="openshift-monitoring/monitoring-plugin-665c9c8584-mjncd"
May 25 11:20:09.506027 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:09.506001    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c3f2ca6f-4b0a-47ae-ae8f-c3fce46b3e8c-monitoring-plugin-cert\") pod \"monitoring-plugin-665c9c8584-mjncd\" (UID: \"c3f2ca6f-4b0a-47ae-ae8f-c3fce46b3e8c\") " pod="openshift-monitoring/monitoring-plugin-665c9c8584-mjncd"
May 25 11:20:09.542711 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:09.542606    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-665c9c8584-mjncd"
May 25 11:20:10.108139 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:20:10.108096    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc39eb3b0_777d_4f3c_b80f_a0b1607419d9.slice/crio-5c349fcec7e29215d6a5cc29b1aba430f10a5f0d4085ad1a84d989f80ddb064f WatchSource:0}: Error finding container 5c349fcec7e29215d6a5cc29b1aba430f10a5f0d4085ad1a84d989f80ddb064f: Status 404 returned error can't find the container with id 5c349fcec7e29215d6a5cc29b1aba430f10a5f0d4085ad1a84d989f80ddb064f
May 25 11:20:10.299613 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:10.299589    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-665c9c8584-mjncd"]
May 25 11:20:10.302836 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:20:10.302813    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3f2ca6f_4b0a_47ae_ae8f_c3fce46b3e8c.slice/crio-59b8aaddc36fc8465536ff05d2fdcb859a441a55a1453575d348642588d6a760 WatchSource:0}: Error finding container 59b8aaddc36fc8465536ff05d2fdcb859a441a55a1453575d348642588d6a760: Status 404 returned error can't find the container with id 59b8aaddc36fc8465536ff05d2fdcb859a441a55a1453575d348642588d6a760
May 25 11:20:10.371902 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:10.371873    2576 patch_prober.go:28] interesting pod/image-registry-6f78b857d8-gcqd2 container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
May 25 11:20:10.371999 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:10.371927    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-6f78b857d8-gcqd2" podUID="6926af5f-04cd-4fed-bb10-669c46bc0c60" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 11:20:10.530809 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:10.530776    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9aa35962-584b-42c7-bb93-a45d4f196267","Type":"ContainerStarted","Data":"7635b3a9e3bdf3a21c995ec5558aabed80377ab635469763f728018512c30a89"}
May 25 11:20:10.530974 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:10.530815    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9aa35962-584b-42c7-bb93-a45d4f196267","Type":"ContainerStarted","Data":"0c4821d509323c13fbe2d38ed630b9ea672577d2c7343dc9b7919c36810ec583"}
May 25 11:20:10.530974 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:10.530831    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9aa35962-584b-42c7-bb93-a45d4f196267","Type":"ContainerStarted","Data":"9d2bb99dcdb215540bf1df11e318a465d1f61df84e1d8916ea2742b58ba3aead"}
May 25 11:20:10.530974 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:10.530842    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9aa35962-584b-42c7-bb93-a45d4f196267","Type":"ContainerStarted","Data":"00a47db1dd0676ba1fc244bde729bcb6bafcf6ac6a09c714965c96984d3935d0"}
May 25 11:20:10.530974 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:10.530851    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9aa35962-584b-42c7-bb93-a45d4f196267","Type":"ContainerStarted","Data":"c4b551a159f02355456b3c977ec2881e7039f3821132a9d7aedd4eb4662ca8df"}
May 25 11:20:10.532529 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:10.532505    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-65f4778c88-nlwjj" event={"ID":"6c5db401-082d-48c0-8393-56392d8afbcd","Type":"ContainerStarted","Data":"2d5b347dbc7fb409908c1a5b5278e43b4d73ee9774448986a64312050a3a4b17"}
May 25 11:20:10.532652 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:10.532534    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-65f4778c88-nlwjj" event={"ID":"6c5db401-082d-48c0-8393-56392d8afbcd","Type":"ContainerStarted","Data":"bc86b9c00802030103b2722db1155c0d7ed695b06f1127aee7c04a0c1373195c"}
May 25 11:20:10.532652 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:10.532543    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-65f4778c88-nlwjj" event={"ID":"6c5db401-082d-48c0-8393-56392d8afbcd","Type":"ContainerStarted","Data":"a4e603f737bd5cd50fef1f32ad7964ad4b5436fffe23c5397f8c86710ed2a754"}
May 25 11:20:10.533384 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:10.533364    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-665c9c8584-mjncd" event={"ID":"c3f2ca6f-4b0a-47ae-ae8f-c3fce46b3e8c","Type":"ContainerStarted","Data":"59b8aaddc36fc8465536ff05d2fdcb859a441a55a1453575d348642588d6a760"}
May 25 11:20:10.534271 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:10.534252    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-84cc7cb899-fs5c9" event={"ID":"c39eb3b0-777d-4f3c-b80f-a0b1607419d9","Type":"ContainerStarted","Data":"5c349fcec7e29215d6a5cc29b1aba430f10a5f0d4085ad1a84d989f80ddb064f"}
May 25 11:20:10.654472 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:10.654400    2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
May 25 11:20:10.658346 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:10.658323    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
May 25 11:20:10.660982 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:10.660957    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
May 25 11:20:10.661099 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:10.661038    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
May 25 11:20:10.661099 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:10.661062    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
May 25 11:20:10.661310 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:10.661294    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-nvblc\""
May 25 11:20:10.661356 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:10.661324    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
May 25 11:20:10.661744 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:10.661723    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
May 25 11:20:10.661854 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:10.661764    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
May 25 11:20:10.661854 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:10.661772    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-bobbetstr1scc\""
May 25 11:20:10.661854 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:10.661821    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
May 25 11:20:10.662020 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:10.661729    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
May 25 11:20:10.662191 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:10.662171    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
May 25 11:20:10.667651 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:10.667375    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
May 25 11:20:10.667651 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:10.667447    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
May 25 11:20:10.673660 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:10.673632    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
May 25 11:20:10.682476 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:10.682449    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
May 25 11:20:10.715921 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:10.715887    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dced9206-c2e9-4a0d-9a21-4e5c898dcfa8-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"dced9206-c2e9-4a0d-9a21-4e5c898dcfa8\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 11:20:10.716062 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:10.715942    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/dced9206-c2e9-4a0d-9a21-4e5c898dcfa8-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"dced9206-c2e9-4a0d-9a21-4e5c898dcfa8\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 11:20:10.716062 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:10.715974    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dced9206-c2e9-4a0d-9a21-4e5c898dcfa8-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"dced9206-c2e9-4a0d-9a21-4e5c898dcfa8\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 11:20:10.716062 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:10.716003    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/dced9206-c2e9-4a0d-9a21-4e5c898dcfa8-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"dced9206-c2e9-4a0d-9a21-4e5c898dcfa8\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 11:20:10.716062 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:10.716037    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dced9206-c2e9-4a0d-9a21-4e5c898dcfa8-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"dced9206-c2e9-4a0d-9a21-4e5c898dcfa8\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 11:20:10.716244 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:10.716075    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/dced9206-c2e9-4a0d-9a21-4e5c898dcfa8-web-config\") pod \"prometheus-k8s-0\" (UID: \"dced9206-c2e9-4a0d-9a21-4e5c898dcfa8\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 11:20:10.716244 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:10.716223    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/dced9206-c2e9-4a0d-9a21-4e5c898dcfa8-config\") pod \"prometheus-k8s-0\" (UID: \"dced9206-c2e9-4a0d-9a21-4e5c898dcfa8\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 11:20:10.716375 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:10.716257    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/dced9206-c2e9-4a0d-9a21-4e5c898dcfa8-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"dced9206-c2e9-4a0d-9a21-4e5c898dcfa8\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 11:20:10.716375 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:10.716347    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/dced9206-c2e9-4a0d-9a21-4e5c898dcfa8-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"dced9206-c2e9-4a0d-9a21-4e5c898dcfa8\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 11:20:10.716470 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:10.716386    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/dced9206-c2e9-4a0d-9a21-4e5c898dcfa8-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"dced9206-c2e9-4a0d-9a21-4e5c898dcfa8\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 11:20:10.716470 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:10.716457    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/dced9206-c2e9-4a0d-9a21-4e5c898dcfa8-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"dced9206-c2e9-4a0d-9a21-4e5c898dcfa8\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 11:20:10.716556 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:10.716480    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/dced9206-c2e9-4a0d-9a21-4e5c898dcfa8-config-out\") pod \"prometheus-k8s-0\" (UID: \"dced9206-c2e9-4a0d-9a21-4e5c898dcfa8\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 11:20:10.716556 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:10.716510    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/dced9206-c2e9-4a0d-9a21-4e5c898dcfa8-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"dced9206-c2e9-4a0d-9a21-4e5c898dcfa8\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 11:20:10.716556 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:10.716547    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8f55h\" (UniqueName: \"kubernetes.io/projected/dced9206-c2e9-4a0d-9a21-4e5c898dcfa8-kube-api-access-8f55h\") pod \"prometheus-k8s-0\" (UID: \"dced9206-c2e9-4a0d-9a21-4e5c898dcfa8\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 11:20:10.716685 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:10.716591    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dced9206-c2e9-4a0d-9a21-4e5c898dcfa8-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"dced9206-c2e9-4a0d-9a21-4e5c898dcfa8\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 11:20:10.716685 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:10.716625    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/dced9206-c2e9-4a0d-9a21-4e5c898dcfa8-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"dced9206-c2e9-4a0d-9a21-4e5c898dcfa8\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 11:20:10.716907 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:10.716884    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/dced9206-c2e9-4a0d-9a21-4e5c898dcfa8-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"dced9206-c2e9-4a0d-9a21-4e5c898dcfa8\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 11:20:10.716965 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:10.716923    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/dced9206-c2e9-4a0d-9a21-4e5c898dcfa8-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"dced9206-c2e9-4a0d-9a21-4e5c898dcfa8\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 11:20:10.806591 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:10.806560    2576 patch_prober.go:28] interesting pod/image-registry-685f589447-dtpd9 container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
May 25 11:20:10.806740 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:10.806611    2576 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-685f589447-dtpd9" podUID="e1e82e94-6cf9-4ca9-a606-fe51ed2aaf29" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 11:20:10.818241 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:10.818213    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/dced9206-c2e9-4a0d-9a21-4e5c898dcfa8-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"dced9206-c2e9-4a0d-9a21-4e5c898dcfa8\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 11:20:10.818378 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:10.818255    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/dced9206-c2e9-4a0d-9a21-4e5c898dcfa8-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"dced9206-c2e9-4a0d-9a21-4e5c898dcfa8\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 11:20:10.818378 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:10.818272    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/dced9206-c2e9-4a0d-9a21-4e5c898dcfa8-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"dced9206-c2e9-4a0d-9a21-4e5c898dcfa8\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 11:20:10.818378 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:10.818330    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/dced9206-c2e9-4a0d-9a21-4e5c898dcfa8-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"dced9206-c2e9-4a0d-9a21-4e5c898dcfa8\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 11:20:10.818378 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:10.818346    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/dced9206-c2e9-4a0d-9a21-4e5c898dcfa8-config-out\") pod \"prometheus-k8s-0\" (UID: \"dced9206-c2e9-4a0d-9a21-4e5c898dcfa8\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 11:20:10.818378 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:10.818362    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/dced9206-c2e9-4a0d-9a21-4e5c898dcfa8-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"dced9206-c2e9-4a0d-9a21-4e5c898dcfa8\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 11:20:10.818627 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:10.818388    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-8f55h\" (UniqueName: \"kubernetes.io/projected/dced9206-c2e9-4a0d-9a21-4e5c898dcfa8-kube-api-access-8f55h\") pod \"prometheus-k8s-0\" (UID: \"dced9206-c2e9-4a0d-9a21-4e5c898dcfa8\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 11:20:10.818627 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:10.818418    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dced9206-c2e9-4a0d-9a21-4e5c898dcfa8-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"dced9206-c2e9-4a0d-9a21-4e5c898dcfa8\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 11:20:10.818784 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:10.818753    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/dced9206-c2e9-4a0d-9a21-4e5c898dcfa8-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"dced9206-c2e9-4a0d-9a21-4e5c898dcfa8\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 11:20:10.818911 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:10.818893    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/dced9206-c2e9-4a0d-9a21-4e5c898dcfa8-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"dced9206-c2e9-4a0d-9a21-4e5c898dcfa8\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 11:20:10.818990 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:10.818974    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/dced9206-c2e9-4a0d-9a21-4e5c898dcfa8-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"dced9206-c2e9-4a0d-9a21-4e5c898dcfa8\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 11:20:10.819071 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:10.819036    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/dced9206-c2e9-4a0d-9a21-4e5c898dcfa8-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"dced9206-c2e9-4a0d-9a21-4e5c898dcfa8\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 11:20:10.819140 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:10.819119    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dced9206-c2e9-4a0d-9a21-4e5c898dcfa8-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"dced9206-c2e9-4a0d-9a21-4e5c898dcfa8\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 11:20:10.819270 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:10.819218    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dced9206-c2e9-4a0d-9a21-4e5c898dcfa8-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"dced9206-c2e9-4a0d-9a21-4e5c898dcfa8\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 11:20:10.819270 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:10.819299    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/dced9206-c2e9-4a0d-9a21-4e5c898dcfa8-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"dced9206-c2e9-4a0d-9a21-4e5c898dcfa8\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 11:20:10.819487 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:10.819343    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dced9206-c2e9-4a0d-9a21-4e5c898dcfa8-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"dced9206-c2e9-4a0d-9a21-4e5c898dcfa8\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 11:20:10.819487 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:10.819379    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/dced9206-c2e9-4a0d-9a21-4e5c898dcfa8-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"dced9206-c2e9-4a0d-9a21-4e5c898dcfa8\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 11:20:10.819487 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:10.819412    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dced9206-c2e9-4a0d-9a21-4e5c898dcfa8-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"dced9206-c2e9-4a0d-9a21-4e5c898dcfa8\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 11:20:10.819487 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:10.819455    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/dced9206-c2e9-4a0d-9a21-4e5c898dcfa8-web-config\") pod \"prometheus-k8s-0\" (UID: \"dced9206-c2e9-4a0d-9a21-4e5c898dcfa8\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 11:20:10.819686 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:10.819519    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/dced9206-c2e9-4a0d-9a21-4e5c898dcfa8-config\") pod \"prometheus-k8s-0\" (UID: \"dced9206-c2e9-4a0d-9a21-4e5c898dcfa8\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 11:20:10.820709 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:10.820064    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dced9206-c2e9-4a0d-9a21-4e5c898dcfa8-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"dced9206-c2e9-4a0d-9a21-4e5c898dcfa8\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 11:20:10.820709 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:10.820414    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dced9206-c2e9-4a0d-9a21-4e5c898dcfa8-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"dced9206-c2e9-4a0d-9a21-4e5c898dcfa8\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 11:20:10.821466 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:10.821412    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/dced9206-c2e9-4a0d-9a21-4e5c898dcfa8-config-out\") pod \"prometheus-k8s-0\" (UID: \"dced9206-c2e9-4a0d-9a21-4e5c898dcfa8\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 11:20:10.821659 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:10.821612    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/dced9206-c2e9-4a0d-9a21-4e5c898dcfa8-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"dced9206-c2e9-4a0d-9a21-4e5c898dcfa8\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 11:20:10.821892 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:10.821847    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/dced9206-c2e9-4a0d-9a21-4e5c898dcfa8-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"dced9206-c2e9-4a0d-9a21-4e5c898dcfa8\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 11:20:10.821981 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:10.821938    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/dced9206-c2e9-4a0d-9a21-4e5c898dcfa8-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"dced9206-c2e9-4a0d-9a21-4e5c898dcfa8\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 11:20:10.822080 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:10.822054    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dced9206-c2e9-4a0d-9a21-4e5c898dcfa8-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"dced9206-c2e9-4a0d-9a21-4e5c898dcfa8\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 11:20:10.822874 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:10.822832    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/dced9206-c2e9-4a0d-9a21-4e5c898dcfa8-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"dced9206-c2e9-4a0d-9a21-4e5c898dcfa8\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 11:20:10.823330 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:10.823307    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/dced9206-c2e9-4a0d-9a21-4e5c898dcfa8-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"dced9206-c2e9-4a0d-9a21-4e5c898dcfa8\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 11:20:10.823474 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:10.823455    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/dced9206-c2e9-4a0d-9a21-4e5c898dcfa8-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"dced9206-c2e9-4a0d-9a21-4e5c898dcfa8\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 11:20:10.823837 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:10.823809    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/dced9206-c2e9-4a0d-9a21-4e5c898dcfa8-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"dced9206-c2e9-4a0d-9a21-4e5c898dcfa8\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 11:20:10.824043 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:10.824023    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/dced9206-c2e9-4a0d-9a21-4e5c898dcfa8-web-config\") pod \"prometheus-k8s-0\" (UID: \"dced9206-c2e9-4a0d-9a21-4e5c898dcfa8\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 11:20:10.824129 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:10.824114    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/dced9206-c2e9-4a0d-9a21-4e5c898dcfa8-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"dced9206-c2e9-4a0d-9a21-4e5c898dcfa8\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 11:20:10.824263 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:10.824244    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/dced9206-c2e9-4a0d-9a21-4e5c898dcfa8-config\") pod \"prometheus-k8s-0\" (UID: \"dced9206-c2e9-4a0d-9a21-4e5c898dcfa8\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 11:20:10.825302 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:10.825263    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/dced9206-c2e9-4a0d-9a21-4e5c898dcfa8-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"dced9206-c2e9-4a0d-9a21-4e5c898dcfa8\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 11:20:10.829491 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:10.829472    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8f55h\" (UniqueName: \"kubernetes.io/projected/dced9206-c2e9-4a0d-9a21-4e5c898dcfa8-kube-api-access-8f55h\") pod \"prometheus-k8s-0\" (UID: \"dced9206-c2e9-4a0d-9a21-4e5c898dcfa8\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 11:20:10.969617 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:10.969578    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
May 25 11:20:11.155917 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:11.155866    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
May 25 11:20:11.230021 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:20:11.229937    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddced9206_c2e9_4a0d_9a21_4e5c898dcfa8.slice/crio-de9b23ff4aa2a941394f49dbc033d3ce0d211570bbb5c5aaa262b42c65a57458 WatchSource:0}: Error finding container de9b23ff4aa2a941394f49dbc033d3ce0d211570bbb5c5aaa262b42c65a57458: Status 404 returned error can't find the container with id de9b23ff4aa2a941394f49dbc033d3ce0d211570bbb5c5aaa262b42c65a57458
May 25 11:20:11.541012 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:11.540899    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"dced9206-c2e9-4a0d-9a21-4e5c898dcfa8","Type":"ContainerStarted","Data":"6230df5b3509ac11cd9e912e5eb0306e5c493e1f86b8fa12e6aca56a4d789af9"}
May 25 11:20:11.541012 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:11.540950    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"dced9206-c2e9-4a0d-9a21-4e5c898dcfa8","Type":"ContainerStarted","Data":"de9b23ff4aa2a941394f49dbc033d3ce0d211570bbb5c5aaa262b42c65a57458"}
May 25 11:20:11.884256 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:20:11.884163    2576 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-8lxgp" podUID="62aaed5e-f12a-4833-acbc-4ea8bfe20fb1"
May 25 11:20:11.898293 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:20:11.898249    2576 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[cert], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-ingress-canary/ingress-canary-cpqmb" podUID="c25259af-1ae3-465a-a802-2dd6fe637e06"
May 25 11:20:12.464162 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:12.464135    2576 patch_prober.go:28] interesting pod/image-registry-685f589447-dtpd9 container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
May 25 11:20:12.464345 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:12.464186    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-685f589447-dtpd9" podUID="e1e82e94-6cf9-4ca9-a606-fe51ed2aaf29" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 11:20:12.548242 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:12.548215    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-665c9c8584-mjncd" event={"ID":"c3f2ca6f-4b0a-47ae-ae8f-c3fce46b3e8c","Type":"ContainerStarted","Data":"8532f4c7001a959cc4cfcea967ee4bd441e98fe55cbc6fc969f002dc4fa2fc45"}
May 25 11:20:12.548498 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:12.548474    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-665c9c8584-mjncd"
May 25 11:20:12.549705 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:12.549682    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-84cc7cb899-fs5c9" event={"ID":"c39eb3b0-777d-4f3c-b80f-a0b1607419d9","Type":"ContainerStarted","Data":"9d8766532f3742de0850cf7b405b3798e57e4d23346923b7738ced5370b8980c"}
May 25 11:20:12.552648 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:12.552626    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9aa35962-584b-42c7-bb93-a45d4f196267","Type":"ContainerStarted","Data":"4ae343b8faec08d9233b61e483b0beffa34af105497ca4e03e147027d1cfa6c6"}
May 25 11:20:12.553941 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:12.553921    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-665c9c8584-mjncd"
May 25 11:20:12.555209 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:12.555189    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-65f4778c88-nlwjj" event={"ID":"6c5db401-082d-48c0-8393-56392d8afbcd","Type":"ContainerStarted","Data":"419a59278303561308b4ee8434699aa794bce9ffbdfc929672855ac292118723"}
May 25 11:20:12.555320 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:12.555213    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-65f4778c88-nlwjj" event={"ID":"6c5db401-082d-48c0-8393-56392d8afbcd","Type":"ContainerStarted","Data":"e8c0c8b090ee03da03a3a00f5f890e781baa148c3b5ea635c35a73f22b675994"}
May 25 11:20:12.555320 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:12.555227    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-65f4778c88-nlwjj" event={"ID":"6c5db401-082d-48c0-8393-56392d8afbcd","Type":"ContainerStarted","Data":"84b7958f192c73ea4d8deda67bfe4594924ee54c32cca1a46977a927ccecdb2a"}
May 25 11:20:12.555433 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:12.555419    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-65f4778c88-nlwjj"
May 25 11:20:12.556447 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:12.556425    2576 generic.go:358] "Generic (PLEG): container finished" podID="dced9206-c2e9-4a0d-9a21-4e5c898dcfa8" containerID="6230df5b3509ac11cd9e912e5eb0306e5c493e1f86b8fa12e6aca56a4d789af9" exitCode=0
May 25 11:20:12.556540 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:12.556487    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-cpqmb"
May 25 11:20:12.556540 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:12.556504    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"dced9206-c2e9-4a0d-9a21-4e5c898dcfa8","Type":"ContainerDied","Data":"6230df5b3509ac11cd9e912e5eb0306e5c493e1f86b8fa12e6aca56a4d789af9"}
May 25 11:20:12.556714 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:12.556702    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-8lxgp"
May 25 11:20:12.568213 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:12.568122    2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-665c9c8584-mjncd" podStartSLOduration=1.8195211119999999 podStartE2EDuration="3.5681033s" podCreationTimestamp="2026-05-25 11:20:09 +0000 UTC" firstStartedPulling="2026-05-25 11:20:10.304786107 +0000 UTC m=+155.822517548" lastFinishedPulling="2026-05-25 11:20:12.053368299 +0000 UTC m=+157.571099736" observedRunningTime="2026-05-25 11:20:12.564941212 +0000 UTC m=+158.082672663" watchObservedRunningTime="2026-05-25 11:20:12.5681033 +0000 UTC m=+158.085834758"
May 25 11:20:12.588619 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:12.588575    2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-65f4778c88-nlwjj" podStartSLOduration=1.530611687 podStartE2EDuration="5.588561894s" podCreationTimestamp="2026-05-25 11:20:07 +0000 UTC" firstStartedPulling="2026-05-25 11:20:07.982042298 +0000 UTC m=+153.499773732" lastFinishedPulling="2026-05-25 11:20:12.0399925 +0000 UTC m=+157.557723939" observedRunningTime="2026-05-25 11:20:12.587518371 +0000 UTC m=+158.105249840" watchObservedRunningTime="2026-05-25 11:20:12.588561894 +0000 UTC m=+158.106293350"
May 25 11:20:12.653904 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:12.653860    2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.956941146 podStartE2EDuration="7.653848092s" podCreationTimestamp="2026-05-25 11:20:05 +0000 UTC" firstStartedPulling="2026-05-25 11:20:06.591937036 +0000 UTC m=+152.109668473" lastFinishedPulling="2026-05-25 11:20:11.28884398 +0000 UTC m=+156.806575419" observedRunningTime="2026-05-25 11:20:12.652469329 +0000 UTC m=+158.170200786" watchObservedRunningTime="2026-05-25 11:20:12.653848092 +0000 UTC m=+158.171579547"
May 25 11:20:12.671736 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:12.671661    2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-84cc7cb899-fs5c9" podStartSLOduration=2.774450414 podStartE2EDuration="4.671645682s" podCreationTimestamp="2026-05-25 11:20:08 +0000 UTC" firstStartedPulling="2026-05-25 11:20:10.150716649 +0000 UTC m=+155.668448083" lastFinishedPulling="2026-05-25 11:20:12.047911895 +0000 UTC m=+157.565643351" observedRunningTime="2026-05-25 11:20:12.669655841 +0000 UTC m=+158.187387297" watchObservedRunningTime="2026-05-25 11:20:12.671645682 +0000 UTC m=+158.189377143"
May 25 11:20:15.385050 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:15.384914    2576 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-image-registry/image-registry-6f78b857d8-gcqd2" podUID="6926af5f-04cd-4fed-bb10-669c46bc0c60" containerName="registry" containerID="cri-o://f2266cabae3f1d3c5a3347837c6262b0b300f8ec53797ddc012f2ed4beaa35c7" gracePeriod=30
May 25 11:20:15.576466 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:15.576419    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"dced9206-c2e9-4a0d-9a21-4e5c898dcfa8","Type":"ContainerStarted","Data":"87f910240b3a5616eadc7d66e802e3e220ba4490cb29fb88164ac6cb97be4f78"}
May 25 11:20:15.576576 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:15.576468    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"dced9206-c2e9-4a0d-9a21-4e5c898dcfa8","Type":"ContainerStarted","Data":"847b784e0a2b0c6885cc1bbfc1ec0def50bde7f32ffa03e3faae57fae770585e"}
May 25 11:20:15.576576 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:15.576486    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"dced9206-c2e9-4a0d-9a21-4e5c898dcfa8","Type":"ContainerStarted","Data":"778d1fbfdd0cc431dc424cd5652fbade69c009d542c55ef45ebd318d04da8be4"}
May 25 11:20:15.578130 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:15.578073    2576 generic.go:358] "Generic (PLEG): container finished" podID="6926af5f-04cd-4fed-bb10-669c46bc0c60" containerID="f2266cabae3f1d3c5a3347837c6262b0b300f8ec53797ddc012f2ed4beaa35c7" exitCode=0
May 25 11:20:15.578130 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:15.578124    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6f78b857d8-gcqd2" event={"ID":"6926af5f-04cd-4fed-bb10-669c46bc0c60","Type":"ContainerDied","Data":"f2266cabae3f1d3c5a3347837c6262b0b300f8ec53797ddc012f2ed4beaa35c7"}
May 25 11:20:15.627543 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:15.627001    2576 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6f78b857d8-gcqd2"
May 25 11:20:15.667865 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:15.667838    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d4r7f\" (UniqueName: \"kubernetes.io/projected/6926af5f-04cd-4fed-bb10-669c46bc0c60-kube-api-access-d4r7f\") pod \"6926af5f-04cd-4fed-bb10-669c46bc0c60\" (UID: \"6926af5f-04cd-4fed-bb10-669c46bc0c60\") "
May 25 11:20:15.667980 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:15.667912    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6926af5f-04cd-4fed-bb10-669c46bc0c60-registry-tls\") pod \"6926af5f-04cd-4fed-bb10-669c46bc0c60\" (UID: \"6926af5f-04cd-4fed-bb10-669c46bc0c60\") "
May 25 11:20:15.667980 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:15.667948    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6926af5f-04cd-4fed-bb10-669c46bc0c60-ca-trust-extracted\") pod \"6926af5f-04cd-4fed-bb10-669c46bc0c60\" (UID: \"6926af5f-04cd-4fed-bb10-669c46bc0c60\") "
May 25 11:20:15.668104 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:15.667988    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6926af5f-04cd-4fed-bb10-669c46bc0c60-trusted-ca\") pod \"6926af5f-04cd-4fed-bb10-669c46bc0c60\" (UID: \"6926af5f-04cd-4fed-bb10-669c46bc0c60\") "
May 25 11:20:15.668104 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:15.668025    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6926af5f-04cd-4fed-bb10-669c46bc0c60-installation-pull-secrets\") pod \"6926af5f-04cd-4fed-bb10-669c46bc0c60\" (UID: \"6926af5f-04cd-4fed-bb10-669c46bc0c60\") "
May 25 11:20:15.668104 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:15.668075    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6926af5f-04cd-4fed-bb10-669c46bc0c60-registry-certificates\") pod \"6926af5f-04cd-4fed-bb10-669c46bc0c60\" (UID: \"6926af5f-04cd-4fed-bb10-669c46bc0c60\") "
May 25 11:20:15.668255 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:15.668146    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6926af5f-04cd-4fed-bb10-669c46bc0c60-bound-sa-token\") pod \"6926af5f-04cd-4fed-bb10-669c46bc0c60\" (UID: \"6926af5f-04cd-4fed-bb10-669c46bc0c60\") "
May 25 11:20:15.668255 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:15.668185    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6926af5f-04cd-4fed-bb10-669c46bc0c60-image-registry-private-configuration\") pod \"6926af5f-04cd-4fed-bb10-669c46bc0c60\" (UID: \"6926af5f-04cd-4fed-bb10-669c46bc0c60\") "
May 25 11:20:15.669843 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:15.668643    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6926af5f-04cd-4fed-bb10-669c46bc0c60-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "6926af5f-04cd-4fed-bb10-669c46bc0c60" (UID: "6926af5f-04cd-4fed-bb10-669c46bc0c60"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 11:20:15.669843 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:15.668838    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6926af5f-04cd-4fed-bb10-669c46bc0c60-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "6926af5f-04cd-4fed-bb10-669c46bc0c60" (UID: "6926af5f-04cd-4fed-bb10-669c46bc0c60"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 11:20:15.672074 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:15.672012    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6926af5f-04cd-4fed-bb10-669c46bc0c60-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "6926af5f-04cd-4fed-bb10-669c46bc0c60" (UID: "6926af5f-04cd-4fed-bb10-669c46bc0c60"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 11:20:15.672549 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:15.672504    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6926af5f-04cd-4fed-bb10-669c46bc0c60-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "6926af5f-04cd-4fed-bb10-669c46bc0c60" (UID: "6926af5f-04cd-4fed-bb10-669c46bc0c60"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 11:20:15.672941 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:15.672909    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6926af5f-04cd-4fed-bb10-669c46bc0c60-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "6926af5f-04cd-4fed-bb10-669c46bc0c60" (UID: "6926af5f-04cd-4fed-bb10-669c46bc0c60"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 11:20:15.673786 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:15.673750    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6926af5f-04cd-4fed-bb10-669c46bc0c60-kube-api-access-d4r7f" (OuterVolumeSpecName: "kube-api-access-d4r7f") pod "6926af5f-04cd-4fed-bb10-669c46bc0c60" (UID: "6926af5f-04cd-4fed-bb10-669c46bc0c60"). InnerVolumeSpecName "kube-api-access-d4r7f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 11:20:15.674247 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:15.674224    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6926af5f-04cd-4fed-bb10-669c46bc0c60-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "6926af5f-04cd-4fed-bb10-669c46bc0c60" (UID: "6926af5f-04cd-4fed-bb10-669c46bc0c60"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 11:20:15.680402 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:15.680379    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6926af5f-04cd-4fed-bb10-669c46bc0c60-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "6926af5f-04cd-4fed-bb10-669c46bc0c60" (UID: "6926af5f-04cd-4fed-bb10-669c46bc0c60"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:20:15.769126 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:15.769102    2576 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6926af5f-04cd-4fed-bb10-669c46bc0c60-image-registry-private-configuration\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
May 25 11:20:15.769126 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:15.769127    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d4r7f\" (UniqueName: \"kubernetes.io/projected/6926af5f-04cd-4fed-bb10-669c46bc0c60-kube-api-access-d4r7f\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
May 25 11:20:15.769255 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:15.769139    2576 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6926af5f-04cd-4fed-bb10-669c46bc0c60-registry-tls\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
May 25 11:20:15.769255 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:15.769148    2576 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6926af5f-04cd-4fed-bb10-669c46bc0c60-ca-trust-extracted\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
May 25 11:20:15.769255 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:15.769162    2576 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6926af5f-04cd-4fed-bb10-669c46bc0c60-trusted-ca\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
May 25 11:20:15.769255 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:15.769173    2576 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6926af5f-04cd-4fed-bb10-669c46bc0c60-installation-pull-secrets\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
May 25 11:20:15.769255 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:15.769181    2576 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6926af5f-04cd-4fed-bb10-669c46bc0c60-registry-certificates\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
May 25 11:20:15.769255 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:15.769190    2576 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6926af5f-04cd-4fed-bb10-669c46bc0c60-bound-sa-token\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
May 25 11:20:16.583940 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:16.583851    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"dced9206-c2e9-4a0d-9a21-4e5c898dcfa8","Type":"ContainerStarted","Data":"12cfb840a9340b7039ecb9d73b1e0d48b8ada060d53ea1591074812b161843f8"}
May 25 11:20:16.583940 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:16.583888    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"dced9206-c2e9-4a0d-9a21-4e5c898dcfa8","Type":"ContainerStarted","Data":"88604427b05b40cf1e75d8da69d674c340583d68f6d0cc780f3544a10f20a8db"}
May 25 11:20:16.583940 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:16.583900    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"dced9206-c2e9-4a0d-9a21-4e5c898dcfa8","Type":"ContainerStarted","Data":"4a672a4b0c61e55e51c4947a70792e48e888143bb9743f0e1ddcd00ea2bac6e2"}
May 25 11:20:16.589574 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:16.589541    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6f78b857d8-gcqd2" event={"ID":"6926af5f-04cd-4fed-bb10-669c46bc0c60","Type":"ContainerDied","Data":"7afe7219e1a04300ef4c4942d3d7cdbfed7cd9423cc1a6e478c1909ffd5ec059"}
May 25 11:20:16.589738 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:16.589591    2576 scope.go:117] "RemoveContainer" containerID="f2266cabae3f1d3c5a3347837c6262b0b300f8ec53797ddc012f2ed4beaa35c7"
May 25 11:20:16.589738 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:16.589607    2576 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6f78b857d8-gcqd2"
May 25 11:20:16.613294 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:16.613233    2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.912688508 podStartE2EDuration="6.613222773s" podCreationTimestamp="2026-05-25 11:20:10 +0000 UTC" firstStartedPulling="2026-05-25 11:20:12.557730459 +0000 UTC m=+158.075461893" lastFinishedPulling="2026-05-25 11:20:15.258264725 +0000 UTC m=+160.775996158" observedRunningTime="2026-05-25 11:20:16.612075548 +0000 UTC m=+162.129807003" watchObservedRunningTime="2026-05-25 11:20:16.613222773 +0000 UTC m=+162.130954230"
May 25 11:20:16.628688 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:16.628667    2576 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6f78b857d8-gcqd2"]
May 25 11:20:16.630555 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:16.630534    2576 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-6f78b857d8-gcqd2"]
May 25 11:20:16.777786 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:16.777756    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/62aaed5e-f12a-4833-acbc-4ea8bfe20fb1-metrics-tls\") pod \"dns-default-8lxgp\" (UID: \"62aaed5e-f12a-4833-acbc-4ea8bfe20fb1\") " pod="openshift-dns/dns-default-8lxgp"
May 25 11:20:16.779924 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:16.779900    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/62aaed5e-f12a-4833-acbc-4ea8bfe20fb1-metrics-tls\") pod \"dns-default-8lxgp\" (UID: \"62aaed5e-f12a-4833-acbc-4ea8bfe20fb1\") " pod="openshift-dns/dns-default-8lxgp"
May 25 11:20:16.879106 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:16.879030    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c25259af-1ae3-465a-a802-2dd6fe637e06-cert\") pod \"ingress-canary-cpqmb\" (UID: \"c25259af-1ae3-465a-a802-2dd6fe637e06\") " pod="openshift-ingress-canary/ingress-canary-cpqmb"
May 25 11:20:16.881300 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:16.881250    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c25259af-1ae3-465a-a802-2dd6fe637e06-cert\") pod \"ingress-canary-cpqmb\" (UID: \"c25259af-1ae3-465a-a802-2dd6fe637e06\") " pod="openshift-ingress-canary/ingress-canary-cpqmb"
May 25 11:20:17.059920 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:17.059889    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-t8nfn\""
May 25 11:20:17.060053 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:17.060033    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-mjf4k\""
May 25 11:20:17.067875 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:17.067859    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-cpqmb"
May 25 11:20:17.067977 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:17.067963    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-8lxgp"
May 25 11:20:17.098749 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:17.098688    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6926af5f-04cd-4fed-bb10-669c46bc0c60" path="/var/lib/kubelet/pods/6926af5f-04cd-4fed-bb10-669c46bc0c60/volumes"
May 25 11:20:17.197856 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:17.197833    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-8lxgp"]
May 25 11:20:17.200035 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:20:17.200006    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62aaed5e_f12a_4833_acbc_4ea8bfe20fb1.slice/crio-90fe81d875aff2c64062a8c1477e120631617058da5fd4c8f75cfa5bd382dbea WatchSource:0}: Error finding container 90fe81d875aff2c64062a8c1477e120631617058da5fd4c8f75cfa5bd382dbea: Status 404 returned error can't find the container with id 90fe81d875aff2c64062a8c1477e120631617058da5fd4c8f75cfa5bd382dbea
May 25 11:20:17.219215 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:17.219187    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-cpqmb"]
May 25 11:20:17.221787 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:20:17.221759    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc25259af_1ae3_465a_a802_2dd6fe637e06.slice/crio-18b4582712fc20922344e6cc1d3e0c004cdaa63835b7a443af376853808bed13 WatchSource:0}: Error finding container 18b4582712fc20922344e6cc1d3e0c004cdaa63835b7a443af376853808bed13: Status 404 returned error can't find the container with id 18b4582712fc20922344e6cc1d3e0c004cdaa63835b7a443af376853808bed13
May 25 11:20:17.594220 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:17.594186    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-cpqmb" event={"ID":"c25259af-1ae3-465a-a802-2dd6fe637e06","Type":"ContainerStarted","Data":"18b4582712fc20922344e6cc1d3e0c004cdaa63835b7a443af376853808bed13"}
May 25 11:20:17.595073 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:17.595045    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-8lxgp" event={"ID":"62aaed5e-f12a-4833-acbc-4ea8bfe20fb1","Type":"ContainerStarted","Data":"90fe81d875aff2c64062a8c1477e120631617058da5fd4c8f75cfa5bd382dbea"}
May 25 11:20:18.566341 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:18.566314    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-65f4778c88-nlwjj"
May 25 11:20:19.605035 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:19.604951    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-cpqmb" event={"ID":"c25259af-1ae3-465a-a802-2dd6fe637e06","Type":"ContainerStarted","Data":"2b07ac15f9cd239ab1ad3e39a2b7fba8d79bcd70a17f63c5bfa4e630c11c1f05"}
May 25 11:20:19.606532 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:19.606508    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-8lxgp" event={"ID":"62aaed5e-f12a-4833-acbc-4ea8bfe20fb1","Type":"ContainerStarted","Data":"cc9a94de3331cae9377098dca80fb361640cddc5f939fecbff341f06ab3d6752"}
May 25 11:20:19.606532 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:19.606535    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-8lxgp" event={"ID":"62aaed5e-f12a-4833-acbc-4ea8bfe20fb1","Type":"ContainerStarted","Data":"ac8ffac66adab1a39b6cfa5b3e7810ae1b786118b2d9d2e30570b50b3ad5142f"}
May 25 11:20:19.606661 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:19.606611    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-8lxgp"
May 25 11:20:19.623286 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:19.623228    2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-cpqmb" podStartSLOduration=129.640232059 podStartE2EDuration="2m11.623213011s" podCreationTimestamp="2026-05-25 11:18:08 +0000 UTC" firstStartedPulling="2026-05-25 11:20:17.223510196 +0000 UTC m=+162.741241633" lastFinishedPulling="2026-05-25 11:20:19.20649114 +0000 UTC m=+164.724222585" observedRunningTime="2026-05-25 11:20:19.621254699 +0000 UTC m=+165.138986155" watchObservedRunningTime="2026-05-25 11:20:19.623213011 +0000 UTC m=+165.140944467"
May 25 11:20:19.643479 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:19.643438    2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-dns/dns-default-8lxgp" podStartSLOduration=129.642753411 podStartE2EDuration="2m11.643427577s" podCreationTimestamp="2026-05-25 11:18:08 +0000 UTC" firstStartedPulling="2026-05-25 11:20:17.202352815 +0000 UTC m=+162.720084265" lastFinishedPulling="2026-05-25 11:20:19.203026995 +0000 UTC m=+164.720758431" observedRunningTime="2026-05-25 11:20:19.641320888 +0000 UTC m=+165.159052344" watchObservedRunningTime="2026-05-25 11:20:19.643427577 +0000 UTC m=+165.161159032"
May 25 11:20:20.805871 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:20.805842    2576 patch_prober.go:28] interesting pod/image-registry-685f589447-dtpd9 container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
May 25 11:20:20.806235 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:20.805890    2576 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-685f589447-dtpd9" podUID="e1e82e94-6cf9-4ca9-a606-fe51ed2aaf29" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 11:20:20.970462 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:20.970429    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
May 25 11:20:22.464967 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:22.464380    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-685f589447-dtpd9"
May 25 11:20:29.195202 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:29.195161    2576 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-84cc7cb899-fs5c9"
May 25 11:20:29.195565 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:29.195231    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-84cc7cb899-fs5c9"
May 25 11:20:29.612142 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:29.612064    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-8lxgp"
May 25 11:20:49.200200 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:49.200171    2576 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-84cc7cb899-fs5c9"
May 25 11:20:49.204010 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:20:49.203989    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-84cc7cb899-fs5c9"
May 25 11:21:10.969996 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:10.969963    2576 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
May 25 11:21:10.985327 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:10.985304    2576 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
May 25 11:21:11.774337 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:11.774313    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
May 25 11:21:24.827895 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:24.827858    2576 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
May 25 11:21:24.828505 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:24.828443    2576 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="9aa35962-584b-42c7-bb93-a45d4f196267" containerName="alertmanager" containerID="cri-o://c4b551a159f02355456b3c977ec2881e7039f3821132a9d7aedd4eb4662ca8df" gracePeriod=120
May 25 11:21:24.828590 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:24.828502    2576 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="9aa35962-584b-42c7-bb93-a45d4f196267" containerName="kube-rbac-proxy-web" containerID="cri-o://9d2bb99dcdb215540bf1df11e318a465d1f61df84e1d8916ea2742b58ba3aead" gracePeriod=120
May 25 11:21:24.828590 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:24.828558    2576 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="9aa35962-584b-42c7-bb93-a45d4f196267" containerName="prom-label-proxy" containerID="cri-o://4ae343b8faec08d9233b61e483b0beffa34af105497ca4e03e147027d1cfa6c6" gracePeriod=120
May 25 11:21:24.828683 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:24.828549    2576 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="9aa35962-584b-42c7-bb93-a45d4f196267" containerName="config-reloader" containerID="cri-o://00a47db1dd0676ba1fc244bde729bcb6bafcf6ac6a09c714965c96984d3935d0" gracePeriod=120
May 25 11:21:24.828683 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:24.828502    2576 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="9aa35962-584b-42c7-bb93-a45d4f196267" containerName="kube-rbac-proxy-metric" containerID="cri-o://7635b3a9e3bdf3a21c995ec5558aabed80377ab635469763f728018512c30a89" gracePeriod=120
May 25 11:21:24.828683 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:24.828515    2576 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="9aa35962-584b-42c7-bb93-a45d4f196267" containerName="kube-rbac-proxy" containerID="cri-o://0c4821d509323c13fbe2d38ed630b9ea672577d2c7343dc9b7919c36810ec583" gracePeriod=120
May 25 11:21:25.801461 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:25.801428    2576 generic.go:358] "Generic (PLEG): container finished" podID="9aa35962-584b-42c7-bb93-a45d4f196267" containerID="4ae343b8faec08d9233b61e483b0beffa34af105497ca4e03e147027d1cfa6c6" exitCode=0
May 25 11:21:25.801461 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:25.801452    2576 generic.go:358] "Generic (PLEG): container finished" podID="9aa35962-584b-42c7-bb93-a45d4f196267" containerID="0c4821d509323c13fbe2d38ed630b9ea672577d2c7343dc9b7919c36810ec583" exitCode=0
May 25 11:21:25.801461 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:25.801458    2576 generic.go:358] "Generic (PLEG): container finished" podID="9aa35962-584b-42c7-bb93-a45d4f196267" containerID="00a47db1dd0676ba1fc244bde729bcb6bafcf6ac6a09c714965c96984d3935d0" exitCode=0
May 25 11:21:25.801461 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:25.801463    2576 generic.go:358] "Generic (PLEG): container finished" podID="9aa35962-584b-42c7-bb93-a45d4f196267" containerID="c4b551a159f02355456b3c977ec2881e7039f3821132a9d7aedd4eb4662ca8df" exitCode=0
May 25 11:21:25.801714 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:25.801499    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9aa35962-584b-42c7-bb93-a45d4f196267","Type":"ContainerDied","Data":"4ae343b8faec08d9233b61e483b0beffa34af105497ca4e03e147027d1cfa6c6"}
May 25 11:21:25.801714 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:25.801532    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9aa35962-584b-42c7-bb93-a45d4f196267","Type":"ContainerDied","Data":"0c4821d509323c13fbe2d38ed630b9ea672577d2c7343dc9b7919c36810ec583"}
May 25 11:21:25.801714 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:25.801543    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9aa35962-584b-42c7-bb93-a45d4f196267","Type":"ContainerDied","Data":"00a47db1dd0676ba1fc244bde729bcb6bafcf6ac6a09c714965c96984d3935d0"}
May 25 11:21:25.801714 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:25.801551    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9aa35962-584b-42c7-bb93-a45d4f196267","Type":"ContainerDied","Data":"c4b551a159f02355456b3c977ec2881e7039f3821132a9d7aedd4eb4662ca8df"}
May 25 11:21:26.064940 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:26.064919    2576 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
May 25 11:21:26.109867 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:26.109835    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9aa35962-584b-42c7-bb93-a45d4f196267-config-out\") pod \"9aa35962-584b-42c7-bb93-a45d4f196267\" (UID: \"9aa35962-584b-42c7-bb93-a45d4f196267\") "
May 25 11:21:26.109867 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:26.109873    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9aa35962-584b-42c7-bb93-a45d4f196267-secret-alertmanager-kube-rbac-proxy\") pod \"9aa35962-584b-42c7-bb93-a45d4f196267\" (UID: \"9aa35962-584b-42c7-bb93-a45d4f196267\") "
May 25 11:21:26.110081 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:26.109891    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9aa35962-584b-42c7-bb93-a45d4f196267-web-config\") pod \"9aa35962-584b-42c7-bb93-a45d4f196267\" (UID: \"9aa35962-584b-42c7-bb93-a45d4f196267\") "
May 25 11:21:26.110081 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:26.109918    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9aa35962-584b-42c7-bb93-a45d4f196267-metrics-client-ca\") pod \"9aa35962-584b-42c7-bb93-a45d4f196267\" (UID: \"9aa35962-584b-42c7-bb93-a45d4f196267\") "
May 25 11:21:26.110081 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:26.109944    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9aa35962-584b-42c7-bb93-a45d4f196267-secret-alertmanager-kube-rbac-proxy-web\") pod \"9aa35962-584b-42c7-bb93-a45d4f196267\" (UID: \"9aa35962-584b-42c7-bb93-a45d4f196267\") "
May 25 11:21:26.110081 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:26.109975    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/9aa35962-584b-42c7-bb93-a45d4f196267-cluster-tls-config\") pod \"9aa35962-584b-42c7-bb93-a45d4f196267\" (UID: \"9aa35962-584b-42c7-bb93-a45d4f196267\") "
May 25 11:21:26.110081 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:26.109997    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/9aa35962-584b-42c7-bb93-a45d4f196267-config-volume\") pod \"9aa35962-584b-42c7-bb93-a45d4f196267\" (UID: \"9aa35962-584b-42c7-bb93-a45d4f196267\") "
May 25 11:21:26.110081 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:26.110027    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/9aa35962-584b-42c7-bb93-a45d4f196267-secret-alertmanager-kube-rbac-proxy-metric\") pod \"9aa35962-584b-42c7-bb93-a45d4f196267\" (UID: \"9aa35962-584b-42c7-bb93-a45d4f196267\") "
May 25 11:21:26.110081 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:26.110055    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/9aa35962-584b-42c7-bb93-a45d4f196267-alertmanager-main-db\") pod \"9aa35962-584b-42c7-bb93-a45d4f196267\" (UID: \"9aa35962-584b-42c7-bb93-a45d4f196267\") "
May 25 11:21:26.110479 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:26.110113    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7wh6t\" (UniqueName: \"kubernetes.io/projected/9aa35962-584b-42c7-bb93-a45d4f196267-kube-api-access-7wh6t\") pod \"9aa35962-584b-42c7-bb93-a45d4f196267\" (UID: \"9aa35962-584b-42c7-bb93-a45d4f196267\") "
May 25 11:21:26.110479 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:26.110143    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9aa35962-584b-42c7-bb93-a45d4f196267-tls-assets\") pod \"9aa35962-584b-42c7-bb93-a45d4f196267\" (UID: \"9aa35962-584b-42c7-bb93-a45d4f196267\") "
May 25 11:21:26.110479 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:26.110187    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9aa35962-584b-42c7-bb93-a45d4f196267-alertmanager-trusted-ca-bundle\") pod \"9aa35962-584b-42c7-bb93-a45d4f196267\" (UID: \"9aa35962-584b-42c7-bb93-a45d4f196267\") "
May 25 11:21:26.110479 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:26.110232    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/9aa35962-584b-42c7-bb93-a45d4f196267-secret-alertmanager-main-tls\") pod \"9aa35962-584b-42c7-bb93-a45d4f196267\" (UID: \"9aa35962-584b-42c7-bb93-a45d4f196267\") "
May 25 11:21:26.110665 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:26.110618    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9aa35962-584b-42c7-bb93-a45d4f196267-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "9aa35962-584b-42c7-bb93-a45d4f196267" (UID: "9aa35962-584b-42c7-bb93-a45d4f196267"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:21:26.110903 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:26.110871    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9aa35962-584b-42c7-bb93-a45d4f196267-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "9aa35962-584b-42c7-bb93-a45d4f196267" (UID: "9aa35962-584b-42c7-bb93-a45d4f196267"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 11:21:26.111685 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:26.111414    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9aa35962-584b-42c7-bb93-a45d4f196267-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "9aa35962-584b-42c7-bb93-a45d4f196267" (UID: "9aa35962-584b-42c7-bb93-a45d4f196267"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 11:21:26.113980 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:26.113702    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9aa35962-584b-42c7-bb93-a45d4f196267-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "9aa35962-584b-42c7-bb93-a45d4f196267" (UID: "9aa35962-584b-42c7-bb93-a45d4f196267"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 11:21:26.114644 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:26.114183    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9aa35962-584b-42c7-bb93-a45d4f196267-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "9aa35962-584b-42c7-bb93-a45d4f196267" (UID: "9aa35962-584b-42c7-bb93-a45d4f196267"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 11:21:26.114644 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:26.114263    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9aa35962-584b-42c7-bb93-a45d4f196267-config-out" (OuterVolumeSpecName: "config-out") pod "9aa35962-584b-42c7-bb93-a45d4f196267" (UID: "9aa35962-584b-42c7-bb93-a45d4f196267"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:21:26.114644 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:26.114436    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9aa35962-584b-42c7-bb93-a45d4f196267-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "9aa35962-584b-42c7-bb93-a45d4f196267" (UID: "9aa35962-584b-42c7-bb93-a45d4f196267"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 11:21:26.114921 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:26.114891    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9aa35962-584b-42c7-bb93-a45d4f196267-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "9aa35962-584b-42c7-bb93-a45d4f196267" (UID: "9aa35962-584b-42c7-bb93-a45d4f196267"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 11:21:26.115056 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:26.114960    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9aa35962-584b-42c7-bb93-a45d4f196267-config-volume" (OuterVolumeSpecName: "config-volume") pod "9aa35962-584b-42c7-bb93-a45d4f196267" (UID: "9aa35962-584b-42c7-bb93-a45d4f196267"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 11:21:26.115541 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:26.115516    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9aa35962-584b-42c7-bb93-a45d4f196267-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "9aa35962-584b-42c7-bb93-a45d4f196267" (UID: "9aa35962-584b-42c7-bb93-a45d4f196267"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 11:21:26.115704 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:26.115687    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9aa35962-584b-42c7-bb93-a45d4f196267-kube-api-access-7wh6t" (OuterVolumeSpecName: "kube-api-access-7wh6t") pod "9aa35962-584b-42c7-bb93-a45d4f196267" (UID: "9aa35962-584b-42c7-bb93-a45d4f196267"). InnerVolumeSpecName "kube-api-access-7wh6t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 11:21:26.120295 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:26.120251    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9aa35962-584b-42c7-bb93-a45d4f196267-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "9aa35962-584b-42c7-bb93-a45d4f196267" (UID: "9aa35962-584b-42c7-bb93-a45d4f196267"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 11:21:26.126747 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:26.126726    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9aa35962-584b-42c7-bb93-a45d4f196267-web-config" (OuterVolumeSpecName: "web-config") pod "9aa35962-584b-42c7-bb93-a45d4f196267" (UID: "9aa35962-584b-42c7-bb93-a45d4f196267"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 11:21:26.211585 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:26.211546    2576 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9aa35962-584b-42c7-bb93-a45d4f196267-metrics-client-ca\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
May 25 11:21:26.211585 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:26.211577    2576 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9aa35962-584b-42c7-bb93-a45d4f196267-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
May 25 11:21:26.211585 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:26.211588    2576 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/9aa35962-584b-42c7-bb93-a45d4f196267-cluster-tls-config\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
May 25 11:21:26.211803 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:26.211598    2576 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/9aa35962-584b-42c7-bb93-a45d4f196267-config-volume\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
May 25 11:21:26.211803 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:26.211609    2576 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/9aa35962-584b-42c7-bb93-a45d4f196267-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
May 25 11:21:26.211803 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:26.211619    2576 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/9aa35962-584b-42c7-bb93-a45d4f196267-alertmanager-main-db\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
May 25 11:21:26.211803 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:26.211628    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7wh6t\" (UniqueName: \"kubernetes.io/projected/9aa35962-584b-42c7-bb93-a45d4f196267-kube-api-access-7wh6t\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
May 25 11:21:26.211803 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:26.211636    2576 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9aa35962-584b-42c7-bb93-a45d4f196267-tls-assets\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
May 25 11:21:26.211803 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:26.211644    2576 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9aa35962-584b-42c7-bb93-a45d4f196267-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
May 25 11:21:26.211803 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:26.211652    2576 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/9aa35962-584b-42c7-bb93-a45d4f196267-secret-alertmanager-main-tls\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
May 25 11:21:26.211803 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:26.211662    2576 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9aa35962-584b-42c7-bb93-a45d4f196267-config-out\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
May 25 11:21:26.211803 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:26.211675    2576 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9aa35962-584b-42c7-bb93-a45d4f196267-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
May 25 11:21:26.211803 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:26.211687    2576 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9aa35962-584b-42c7-bb93-a45d4f196267-web-config\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
May 25 11:21:26.807075 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:26.807042    2576 generic.go:358] "Generic (PLEG): container finished" podID="9aa35962-584b-42c7-bb93-a45d4f196267" containerID="7635b3a9e3bdf3a21c995ec5558aabed80377ab635469763f728018512c30a89" exitCode=0
May 25 11:21:26.807075 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:26.807067    2576 generic.go:358] "Generic (PLEG): container finished" podID="9aa35962-584b-42c7-bb93-a45d4f196267" containerID="9d2bb99dcdb215540bf1df11e318a465d1f61df84e1d8916ea2742b58ba3aead" exitCode=0
May 25 11:21:26.807364 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:26.807131    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9aa35962-584b-42c7-bb93-a45d4f196267","Type":"ContainerDied","Data":"7635b3a9e3bdf3a21c995ec5558aabed80377ab635469763f728018512c30a89"}
May 25 11:21:26.807364 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:26.807178    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9aa35962-584b-42c7-bb93-a45d4f196267","Type":"ContainerDied","Data":"9d2bb99dcdb215540bf1df11e318a465d1f61df84e1d8916ea2742b58ba3aead"}
May 25 11:21:26.807364 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:26.807192    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9aa35962-584b-42c7-bb93-a45d4f196267","Type":"ContainerDied","Data":"80dbc3b16a4378a3f980291bc8a723b8e83723d6f470b700f70e6b308920fe3b"}
May 25 11:21:26.807364 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:26.807211    2576 scope.go:117] "RemoveContainer" containerID="4ae343b8faec08d9233b61e483b0beffa34af105497ca4e03e147027d1cfa6c6"
May 25 11:21:26.807364 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:26.807142    2576 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
May 25 11:21:26.814982 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:26.814965    2576 scope.go:117] "RemoveContainer" containerID="7635b3a9e3bdf3a21c995ec5558aabed80377ab635469763f728018512c30a89"
May 25 11:21:26.821667 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:26.821648    2576 scope.go:117] "RemoveContainer" containerID="0c4821d509323c13fbe2d38ed630b9ea672577d2c7343dc9b7919c36810ec583"
May 25 11:21:26.828190 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:26.828169    2576 scope.go:117] "RemoveContainer" containerID="9d2bb99dcdb215540bf1df11e318a465d1f61df84e1d8916ea2742b58ba3aead"
May 25 11:21:26.830027 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:26.829937    2576 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
May 25 11:21:26.836264 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:26.836237    2576 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
May 25 11:21:26.836735 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:26.836712    2576 scope.go:117] "RemoveContainer" containerID="00a47db1dd0676ba1fc244bde729bcb6bafcf6ac6a09c714965c96984d3935d0"
May 25 11:21:26.843506 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:26.843489    2576 scope.go:117] "RemoveContainer" containerID="c4b551a159f02355456b3c977ec2881e7039f3821132a9d7aedd4eb4662ca8df"
May 25 11:21:26.849783 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:26.849766    2576 scope.go:117] "RemoveContainer" containerID="11195249a844e7bf61ca71b8650d121ef9bec7e1c30562091b6189bb0998d647"
May 25 11:21:26.856543 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:26.856523    2576 scope.go:117] "RemoveContainer" containerID="4ae343b8faec08d9233b61e483b0beffa34af105497ca4e03e147027d1cfa6c6"
May 25 11:21:26.856799 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:21:26.856780    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ae343b8faec08d9233b61e483b0beffa34af105497ca4e03e147027d1cfa6c6\": container with ID starting with 4ae343b8faec08d9233b61e483b0beffa34af105497ca4e03e147027d1cfa6c6 not found: ID does not exist" containerID="4ae343b8faec08d9233b61e483b0beffa34af105497ca4e03e147027d1cfa6c6"
May 25 11:21:26.856849 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:26.856809    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ae343b8faec08d9233b61e483b0beffa34af105497ca4e03e147027d1cfa6c6"} err="failed to get container status \"4ae343b8faec08d9233b61e483b0beffa34af105497ca4e03e147027d1cfa6c6\": rpc error: code = NotFound desc = could not find container \"4ae343b8faec08d9233b61e483b0beffa34af105497ca4e03e147027d1cfa6c6\": container with ID starting with 4ae343b8faec08d9233b61e483b0beffa34af105497ca4e03e147027d1cfa6c6 not found: ID does not exist"
May 25 11:21:26.856849 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:26.856844    2576 scope.go:117] "RemoveContainer" containerID="7635b3a9e3bdf3a21c995ec5558aabed80377ab635469763f728018512c30a89"
May 25 11:21:26.857078 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:21:26.857064    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7635b3a9e3bdf3a21c995ec5558aabed80377ab635469763f728018512c30a89\": container with ID starting with 7635b3a9e3bdf3a21c995ec5558aabed80377ab635469763f728018512c30a89 not found: ID does not exist" containerID="7635b3a9e3bdf3a21c995ec5558aabed80377ab635469763f728018512c30a89"
May 25 11:21:26.857116 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:26.857081    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7635b3a9e3bdf3a21c995ec5558aabed80377ab635469763f728018512c30a89"} err="failed to get container status \"7635b3a9e3bdf3a21c995ec5558aabed80377ab635469763f728018512c30a89\": rpc error: code = NotFound desc = could not find container \"7635b3a9e3bdf3a21c995ec5558aabed80377ab635469763f728018512c30a89\": container with ID starting with 7635b3a9e3bdf3a21c995ec5558aabed80377ab635469763f728018512c30a89 not found: ID does not exist"
May 25 11:21:26.857116 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:26.857093    2576 scope.go:117] "RemoveContainer" containerID="0c4821d509323c13fbe2d38ed630b9ea672577d2c7343dc9b7919c36810ec583"
May 25 11:21:26.857314 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:21:26.857295    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c4821d509323c13fbe2d38ed630b9ea672577d2c7343dc9b7919c36810ec583\": container with ID starting with 0c4821d509323c13fbe2d38ed630b9ea672577d2c7343dc9b7919c36810ec583 not found: ID does not exist" containerID="0c4821d509323c13fbe2d38ed630b9ea672577d2c7343dc9b7919c36810ec583"
May 25 11:21:26.857406 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:26.857314    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c4821d509323c13fbe2d38ed630b9ea672577d2c7343dc9b7919c36810ec583"} err="failed to get container status \"0c4821d509323c13fbe2d38ed630b9ea672577d2c7343dc9b7919c36810ec583\": rpc error: code = NotFound desc = could not find container \"0c4821d509323c13fbe2d38ed630b9ea672577d2c7343dc9b7919c36810ec583\": container with ID starting with 0c4821d509323c13fbe2d38ed630b9ea672577d2c7343dc9b7919c36810ec583 not found: ID does not exist"
May 25 11:21:26.857406 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:26.857326    2576 scope.go:117] "RemoveContainer" containerID="9d2bb99dcdb215540bf1df11e318a465d1f61df84e1d8916ea2742b58ba3aead"
May 25 11:21:26.857557 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:21:26.857540    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d2bb99dcdb215540bf1df11e318a465d1f61df84e1d8916ea2742b58ba3aead\": container with ID starting with 9d2bb99dcdb215540bf1df11e318a465d1f61df84e1d8916ea2742b58ba3aead not found: ID does not exist" containerID="9d2bb99dcdb215540bf1df11e318a465d1f61df84e1d8916ea2742b58ba3aead"
May 25 11:21:26.857605 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:26.857560    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d2bb99dcdb215540bf1df11e318a465d1f61df84e1d8916ea2742b58ba3aead"} err="failed to get container status \"9d2bb99dcdb215540bf1df11e318a465d1f61df84e1d8916ea2742b58ba3aead\": rpc error: code = NotFound desc = could not find container \"9d2bb99dcdb215540bf1df11e318a465d1f61df84e1d8916ea2742b58ba3aead\": container with ID starting with 9d2bb99dcdb215540bf1df11e318a465d1f61df84e1d8916ea2742b58ba3aead not found: ID does not exist"
May 25 11:21:26.857605 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:26.857573    2576 scope.go:117] "RemoveContainer" containerID="00a47db1dd0676ba1fc244bde729bcb6bafcf6ac6a09c714965c96984d3935d0"
May 25 11:21:26.857780 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:21:26.857766    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00a47db1dd0676ba1fc244bde729bcb6bafcf6ac6a09c714965c96984d3935d0\": container with ID starting with 00a47db1dd0676ba1fc244bde729bcb6bafcf6ac6a09c714965c96984d3935d0 not found: ID does not exist" containerID="00a47db1dd0676ba1fc244bde729bcb6bafcf6ac6a09c714965c96984d3935d0"
May 25 11:21:26.857816 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:26.857782    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00a47db1dd0676ba1fc244bde729bcb6bafcf6ac6a09c714965c96984d3935d0"} err="failed to get container status \"00a47db1dd0676ba1fc244bde729bcb6bafcf6ac6a09c714965c96984d3935d0\": rpc error: code = NotFound desc = could not find container \"00a47db1dd0676ba1fc244bde729bcb6bafcf6ac6a09c714965c96984d3935d0\": container with ID starting with 00a47db1dd0676ba1fc244bde729bcb6bafcf6ac6a09c714965c96984d3935d0 not found: ID does not exist"
May 25 11:21:26.857816 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:26.857797    2576 scope.go:117] "RemoveContainer" containerID="c4b551a159f02355456b3c977ec2881e7039f3821132a9d7aedd4eb4662ca8df"
May 25 11:21:26.858039 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:21:26.858020    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4b551a159f02355456b3c977ec2881e7039f3821132a9d7aedd4eb4662ca8df\": container with ID starting with c4b551a159f02355456b3c977ec2881e7039f3821132a9d7aedd4eb4662ca8df not found: ID does not exist" containerID="c4b551a159f02355456b3c977ec2881e7039f3821132a9d7aedd4eb4662ca8df"
May 25 11:21:26.858100 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:26.858047    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4b551a159f02355456b3c977ec2881e7039f3821132a9d7aedd4eb4662ca8df"} err="failed to get container status \"c4b551a159f02355456b3c977ec2881e7039f3821132a9d7aedd4eb4662ca8df\": rpc error: code = NotFound desc = could not find container \"c4b551a159f02355456b3c977ec2881e7039f3821132a9d7aedd4eb4662ca8df\": container with ID starting with c4b551a159f02355456b3c977ec2881e7039f3821132a9d7aedd4eb4662ca8df not found: ID does not exist"
May 25 11:21:26.858100 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:26.858070    2576 scope.go:117] "RemoveContainer" containerID="11195249a844e7bf61ca71b8650d121ef9bec7e1c30562091b6189bb0998d647"
May 25 11:21:26.858316 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:21:26.858300    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11195249a844e7bf61ca71b8650d121ef9bec7e1c30562091b6189bb0998d647\": container with ID starting with 11195249a844e7bf61ca71b8650d121ef9bec7e1c30562091b6189bb0998d647 not found: ID does not exist" containerID="11195249a844e7bf61ca71b8650d121ef9bec7e1c30562091b6189bb0998d647"
May 25 11:21:26.858365 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:26.858318    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11195249a844e7bf61ca71b8650d121ef9bec7e1c30562091b6189bb0998d647"} err="failed to get container status \"11195249a844e7bf61ca71b8650d121ef9bec7e1c30562091b6189bb0998d647\": rpc error: code = NotFound desc = could not find container \"11195249a844e7bf61ca71b8650d121ef9bec7e1c30562091b6189bb0998d647\": container with ID starting with 11195249a844e7bf61ca71b8650d121ef9bec7e1c30562091b6189bb0998d647 not found: ID does not exist"
May 25 11:21:26.858365 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:26.858330    2576 scope.go:117] "RemoveContainer" containerID="4ae343b8faec08d9233b61e483b0beffa34af105497ca4e03e147027d1cfa6c6"
May 25 11:21:26.858544 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:26.858525    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ae343b8faec08d9233b61e483b0beffa34af105497ca4e03e147027d1cfa6c6"} err="failed to get container status \"4ae343b8faec08d9233b61e483b0beffa34af105497ca4e03e147027d1cfa6c6\": rpc error: code = NotFound desc = could not find container \"4ae343b8faec08d9233b61e483b0beffa34af105497ca4e03e147027d1cfa6c6\": container with ID starting with 4ae343b8faec08d9233b61e483b0beffa34af105497ca4e03e147027d1cfa6c6 not found: ID does not exist"
May 25 11:21:26.858592 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:26.858546    2576 scope.go:117] "RemoveContainer" containerID="7635b3a9e3bdf3a21c995ec5558aabed80377ab635469763f728018512c30a89"
May 25 11:21:26.858759 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:26.858740    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7635b3a9e3bdf3a21c995ec5558aabed80377ab635469763f728018512c30a89"} err="failed to get container status \"7635b3a9e3bdf3a21c995ec5558aabed80377ab635469763f728018512c30a89\": rpc error: code = NotFound desc = could not find container \"7635b3a9e3bdf3a21c995ec5558aabed80377ab635469763f728018512c30a89\": container with ID starting with 7635b3a9e3bdf3a21c995ec5558aabed80377ab635469763f728018512c30a89 not found: ID does not exist"
May 25 11:21:26.858801 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:26.858759    2576 scope.go:117] "RemoveContainer" containerID="0c4821d509323c13fbe2d38ed630b9ea672577d2c7343dc9b7919c36810ec583"
May 25 11:21:26.858982 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:26.858966    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c4821d509323c13fbe2d38ed630b9ea672577d2c7343dc9b7919c36810ec583"} err="failed to get container status \"0c4821d509323c13fbe2d38ed630b9ea672577d2c7343dc9b7919c36810ec583\": rpc error: code = NotFound desc = could not find container \"0c4821d509323c13fbe2d38ed630b9ea672577d2c7343dc9b7919c36810ec583\": container with ID starting with 0c4821d509323c13fbe2d38ed630b9ea672577d2c7343dc9b7919c36810ec583 not found: ID does not exist"
May 25 11:21:26.859041 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:26.858983    2576 scope.go:117] "RemoveContainer" containerID="9d2bb99dcdb215540bf1df11e318a465d1f61df84e1d8916ea2742b58ba3aead"
May 25 11:21:26.859159 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:26.859141    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d2bb99dcdb215540bf1df11e318a465d1f61df84e1d8916ea2742b58ba3aead"} err="failed to get container status \"9d2bb99dcdb215540bf1df11e318a465d1f61df84e1d8916ea2742b58ba3aead\": rpc error: code = NotFound desc = could not find container \"9d2bb99dcdb215540bf1df11e318a465d1f61df84e1d8916ea2742b58ba3aead\": container with ID starting with 9d2bb99dcdb215540bf1df11e318a465d1f61df84e1d8916ea2742b58ba3aead not found: ID does not exist"
May 25 11:21:26.859200 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:26.859161    2576 scope.go:117] "RemoveContainer" containerID="00a47db1dd0676ba1fc244bde729bcb6bafcf6ac6a09c714965c96984d3935d0"
May 25 11:21:26.859361 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:26.859343    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00a47db1dd0676ba1fc244bde729bcb6bafcf6ac6a09c714965c96984d3935d0"} err="failed to get container status \"00a47db1dd0676ba1fc244bde729bcb6bafcf6ac6a09c714965c96984d3935d0\": rpc error: code = NotFound desc = could not find container \"00a47db1dd0676ba1fc244bde729bcb6bafcf6ac6a09c714965c96984d3935d0\": container with ID starting with 00a47db1dd0676ba1fc244bde729bcb6bafcf6ac6a09c714965c96984d3935d0 not found: ID does not exist"
May 25 11:21:26.859361 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:26.859359    2576 scope.go:117] "RemoveContainer" containerID="c4b551a159f02355456b3c977ec2881e7039f3821132a9d7aedd4eb4662ca8df"
May 25 11:21:26.859584 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:26.859568    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4b551a159f02355456b3c977ec2881e7039f3821132a9d7aedd4eb4662ca8df"} err="failed to get container status \"c4b551a159f02355456b3c977ec2881e7039f3821132a9d7aedd4eb4662ca8df\": rpc error: code = NotFound desc = could not find container \"c4b551a159f02355456b3c977ec2881e7039f3821132a9d7aedd4eb4662ca8df\": container with ID starting with c4b551a159f02355456b3c977ec2881e7039f3821132a9d7aedd4eb4662ca8df not found: ID does not exist"
May 25 11:21:26.859633 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:26.859584    2576 scope.go:117] "RemoveContainer" containerID="11195249a844e7bf61ca71b8650d121ef9bec7e1c30562091b6189bb0998d647"
May 25 11:21:26.859777 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:26.859757    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11195249a844e7bf61ca71b8650d121ef9bec7e1c30562091b6189bb0998d647"} err="failed to get container status \"11195249a844e7bf61ca71b8650d121ef9bec7e1c30562091b6189bb0998d647\": rpc error: code = NotFound desc = could not find container \"11195249a844e7bf61ca71b8650d121ef9bec7e1c30562091b6189bb0998d647\": container with ID starting with 11195249a844e7bf61ca71b8650d121ef9bec7e1c30562091b6189bb0998d647 not found: ID does not exist"
May 25 11:21:26.862811 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:26.862790    2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
May 25 11:21:26.863091 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:26.863076    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9aa35962-584b-42c7-bb93-a45d4f196267" containerName="kube-rbac-proxy-metric"
May 25 11:21:26.863133 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:26.863095    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="9aa35962-584b-42c7-bb93-a45d4f196267" containerName="kube-rbac-proxy-metric"
May 25 11:21:26.863133 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:26.863106    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9aa35962-584b-42c7-bb93-a45d4f196267" containerName="kube-rbac-proxy"
May 25 11:21:26.863133 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:26.863111    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="9aa35962-584b-42c7-bb93-a45d4f196267" containerName="kube-rbac-proxy"
May 25 11:21:26.863133 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:26.863122    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9aa35962-584b-42c7-bb93-a45d4f196267" containerName="alertmanager"
May 25 11:21:26.863133 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:26.863127    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="9aa35962-584b-42c7-bb93-a45d4f196267" containerName="alertmanager"
May 25 11:21:26.863133 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:26.863134    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9aa35962-584b-42c7-bb93-a45d4f196267" containerName="prom-label-proxy"
May 25 11:21:26.863324 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:26.863139    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="9aa35962-584b-42c7-bb93-a45d4f196267" containerName="prom-label-proxy"
May 25 11:21:26.863324 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:26.863151    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6926af5f-04cd-4fed-bb10-669c46bc0c60" containerName="registry"
May 25 11:21:26.863324 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:26.863158    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="6926af5f-04cd-4fed-bb10-669c46bc0c60" containerName="registry"
May 25 11:21:26.863324 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:26.863168    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9aa35962-584b-42c7-bb93-a45d4f196267" containerName="config-reloader"
May 25 11:21:26.863324 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:26.863175    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="9aa35962-584b-42c7-bb93-a45d4f196267" containerName="config-reloader"
May 25 11:21:26.863324 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:26.863183    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9aa35962-584b-42c7-bb93-a45d4f196267" containerName="kube-rbac-proxy-web"
May 25 11:21:26.863324 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:26.863188    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="9aa35962-584b-42c7-bb93-a45d4f196267" containerName="kube-rbac-proxy-web"
May 25 11:21:26.863324 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:26.863194    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9aa35962-584b-42c7-bb93-a45d4f196267" containerName="init-config-reloader"
May 25 11:21:26.863324 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:26.863199    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="9aa35962-584b-42c7-bb93-a45d4f196267" containerName="init-config-reloader"
May 25 11:21:26.863324 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:26.863244    2576 memory_manager.go:356] "RemoveStaleState removing state" podUID="9aa35962-584b-42c7-bb93-a45d4f196267" containerName="kube-rbac-proxy"
May 25 11:21:26.863324 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:26.863254    2576 memory_manager.go:356] "RemoveStaleState removing state" podUID="9aa35962-584b-42c7-bb93-a45d4f196267" containerName="kube-rbac-proxy-metric"
May 25 11:21:26.863324 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:26.863262    2576 memory_manager.go:356] "RemoveStaleState removing state" podUID="9aa35962-584b-42c7-bb93-a45d4f196267" containerName="kube-rbac-proxy-web"
May 25 11:21:26.863324 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:26.863269    2576 memory_manager.go:356] "RemoveStaleState removing state" podUID="6926af5f-04cd-4fed-bb10-669c46bc0c60" containerName="registry"
May 25 11:21:26.863324 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:26.863292    2576 memory_manager.go:356] "RemoveStaleState removing state" podUID="9aa35962-584b-42c7-bb93-a45d4f196267" containerName="prom-label-proxy"
May 25 11:21:26.863324 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:26.863303    2576 memory_manager.go:356] "RemoveStaleState removing state" podUID="9aa35962-584b-42c7-bb93-a45d4f196267" containerName="alertmanager"
May 25 11:21:26.863324 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:26.863312    2576 memory_manager.go:356] "RemoveStaleState removing state" podUID="9aa35962-584b-42c7-bb93-a45d4f196267" containerName="config-reloader"
May 25 11:21:26.868187 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:26.868170    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
May 25 11:21:26.870766 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:26.870739    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
May 25 11:21:26.870921 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:26.870830    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
May 25 11:21:26.873682 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:26.871097    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
May 25 11:21:26.873682 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:26.871374    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
May 25 11:21:26.873682 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:26.873151    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
May 25 11:21:26.873682 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:26.873208    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
May 25 11:21:26.873682 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:26.873545    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
May 25 11:21:26.873992 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:26.873849    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
May 25 11:21:26.873992 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:26.873916    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-m7485\""
May 25 11:21:26.880153 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:26.880118    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
May 25 11:21:26.880793 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:26.880768    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
May 25 11:21:26.916857 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:26.916826    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7m6j\" (UniqueName: \"kubernetes.io/projected/a8391b8d-7b7f-4387-864d-63297d47d26e-kube-api-access-k7m6j\") pod \"alertmanager-main-0\" (UID: \"a8391b8d-7b7f-4387-864d-63297d47d26e\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 11:21:26.917038 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:26.916865    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a8391b8d-7b7f-4387-864d-63297d47d26e-tls-assets\") pod \"alertmanager-main-0\" (UID: \"a8391b8d-7b7f-4387-864d-63297d47d26e\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 11:21:26.917038 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:26.916886    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a8391b8d-7b7f-4387-864d-63297d47d26e-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"a8391b8d-7b7f-4387-864d-63297d47d26e\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 11:21:26.917038 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:26.916902    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a8391b8d-7b7f-4387-864d-63297d47d26e-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"a8391b8d-7b7f-4387-864d-63297d47d26e\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 11:21:26.917038 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:26.916918    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a8391b8d-7b7f-4387-864d-63297d47d26e-config-volume\") pod \"alertmanager-main-0\" (UID: \"a8391b8d-7b7f-4387-864d-63297d47d26e\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 11:21:26.917038 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:26.916934    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a8391b8d-7b7f-4387-864d-63297d47d26e-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"a8391b8d-7b7f-4387-864d-63297d47d26e\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 11:21:26.917038 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:26.916951    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a8391b8d-7b7f-4387-864d-63297d47d26e-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"a8391b8d-7b7f-4387-864d-63297d47d26e\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 11:21:26.917038 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:26.916970    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a8391b8d-7b7f-4387-864d-63297d47d26e-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"a8391b8d-7b7f-4387-864d-63297d47d26e\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 11:21:26.917038 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:26.916993    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a8391b8d-7b7f-4387-864d-63297d47d26e-web-config\") pod \"alertmanager-main-0\" (UID: \"a8391b8d-7b7f-4387-864d-63297d47d26e\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 11:21:26.917399 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:26.917059    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/a8391b8d-7b7f-4387-864d-63297d47d26e-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"a8391b8d-7b7f-4387-864d-63297d47d26e\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 11:21:26.917399 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:26.917121    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a8391b8d-7b7f-4387-864d-63297d47d26e-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"a8391b8d-7b7f-4387-864d-63297d47d26e\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 11:21:26.917399 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:26.917163    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a8391b8d-7b7f-4387-864d-63297d47d26e-config-out\") pod \"alertmanager-main-0\" (UID: \"a8391b8d-7b7f-4387-864d-63297d47d26e\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 11:21:26.917399 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:26.917198    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/a8391b8d-7b7f-4387-864d-63297d47d26e-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"a8391b8d-7b7f-4387-864d-63297d47d26e\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 11:21:27.018593 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:27.018559    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-k7m6j\" (UniqueName: \"kubernetes.io/projected/a8391b8d-7b7f-4387-864d-63297d47d26e-kube-api-access-k7m6j\") pod \"alertmanager-main-0\" (UID: \"a8391b8d-7b7f-4387-864d-63297d47d26e\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 11:21:27.018593 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:27.018597    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a8391b8d-7b7f-4387-864d-63297d47d26e-tls-assets\") pod \"alertmanager-main-0\" (UID: \"a8391b8d-7b7f-4387-864d-63297d47d26e\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 11:21:27.018791 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:27.018624    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a8391b8d-7b7f-4387-864d-63297d47d26e-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"a8391b8d-7b7f-4387-864d-63297d47d26e\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 11:21:27.018791 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:27.018744    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a8391b8d-7b7f-4387-864d-63297d47d26e-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"a8391b8d-7b7f-4387-864d-63297d47d26e\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 11:21:27.018791 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:27.018775    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a8391b8d-7b7f-4387-864d-63297d47d26e-config-volume\") pod \"alertmanager-main-0\" (UID: \"a8391b8d-7b7f-4387-864d-63297d47d26e\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 11:21:27.018906 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:27.018795    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a8391b8d-7b7f-4387-864d-63297d47d26e-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"a8391b8d-7b7f-4387-864d-63297d47d26e\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 11:21:27.018906 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:27.018814    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a8391b8d-7b7f-4387-864d-63297d47d26e-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"a8391b8d-7b7f-4387-864d-63297d47d26e\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 11:21:27.018906 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:27.018850    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a8391b8d-7b7f-4387-864d-63297d47d26e-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"a8391b8d-7b7f-4387-864d-63297d47d26e\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 11:21:27.018906 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:27.018894    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a8391b8d-7b7f-4387-864d-63297d47d26e-web-config\") pod \"alertmanager-main-0\" (UID: \"a8391b8d-7b7f-4387-864d-63297d47d26e\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 11:21:27.019094 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:27.018925    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/a8391b8d-7b7f-4387-864d-63297d47d26e-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"a8391b8d-7b7f-4387-864d-63297d47d26e\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 11:21:27.019094 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:27.018962    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a8391b8d-7b7f-4387-864d-63297d47d26e-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"a8391b8d-7b7f-4387-864d-63297d47d26e\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 11:21:27.019094 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:27.018996    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a8391b8d-7b7f-4387-864d-63297d47d26e-config-out\") pod \"alertmanager-main-0\" (UID: \"a8391b8d-7b7f-4387-864d-63297d47d26e\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 11:21:27.019094 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:27.019027    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/a8391b8d-7b7f-4387-864d-63297d47d26e-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"a8391b8d-7b7f-4387-864d-63297d47d26e\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 11:21:27.019562 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:27.019535    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/a8391b8d-7b7f-4387-864d-63297d47d26e-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"a8391b8d-7b7f-4387-864d-63297d47d26e\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 11:21:27.020089 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:27.020069    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a8391b8d-7b7f-4387-864d-63297d47d26e-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"a8391b8d-7b7f-4387-864d-63297d47d26e\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 11:21:27.020227 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:27.020069    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a8391b8d-7b7f-4387-864d-63297d47d26e-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"a8391b8d-7b7f-4387-864d-63297d47d26e\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 11:21:27.021703 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:27.021674    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/a8391b8d-7b7f-4387-864d-63297d47d26e-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"a8391b8d-7b7f-4387-864d-63297d47d26e\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 11:21:27.021802 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:27.021757    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a8391b8d-7b7f-4387-864d-63297d47d26e-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"a8391b8d-7b7f-4387-864d-63297d47d26e\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 11:21:27.021802 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:27.021782    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a8391b8d-7b7f-4387-864d-63297d47d26e-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"a8391b8d-7b7f-4387-864d-63297d47d26e\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 11:21:27.021977 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:27.021950    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a8391b8d-7b7f-4387-864d-63297d47d26e-config-volume\") pod \"alertmanager-main-0\" (UID: \"a8391b8d-7b7f-4387-864d-63297d47d26e\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 11:21:27.022095 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:27.022064    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a8391b8d-7b7f-4387-864d-63297d47d26e-web-config\") pod \"alertmanager-main-0\" (UID: \"a8391b8d-7b7f-4387-864d-63297d47d26e\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 11:21:27.022154 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:27.022097    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a8391b8d-7b7f-4387-864d-63297d47d26e-tls-assets\") pod \"alertmanager-main-0\" (UID: \"a8391b8d-7b7f-4387-864d-63297d47d26e\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 11:21:27.022664 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:27.022643    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a8391b8d-7b7f-4387-864d-63297d47d26e-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"a8391b8d-7b7f-4387-864d-63297d47d26e\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 11:21:27.022765 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:27.022746    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a8391b8d-7b7f-4387-864d-63297d47d26e-config-out\") pod \"alertmanager-main-0\" (UID: \"a8391b8d-7b7f-4387-864d-63297d47d26e\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 11:21:27.023782 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:27.023766    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a8391b8d-7b7f-4387-864d-63297d47d26e-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"a8391b8d-7b7f-4387-864d-63297d47d26e\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 11:21:27.026932 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:27.026911    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7m6j\" (UniqueName: \"kubernetes.io/projected/a8391b8d-7b7f-4387-864d-63297d47d26e-kube-api-access-k7m6j\") pod \"alertmanager-main-0\" (UID: \"a8391b8d-7b7f-4387-864d-63297d47d26e\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 11:21:27.097790 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:27.097715    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9aa35962-584b-42c7-bb93-a45d4f196267" path="/var/lib/kubelet/pods/9aa35962-584b-42c7-bb93-a45d4f196267/volumes"
May 25 11:21:27.191205 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:27.191176    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
May 25 11:21:27.323175 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:27.323052    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
May 25 11:21:27.325618 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:21:27.325591    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8391b8d_7b7f_4387_864d_63297d47d26e.slice/crio-62668ab23acb3fed67d0673c020ed57457ca0cc6769dab3caa9d1771ed6567a3 WatchSource:0}: Error finding container 62668ab23acb3fed67d0673c020ed57457ca0cc6769dab3caa9d1771ed6567a3: Status 404 returned error can't find the container with id 62668ab23acb3fed67d0673c020ed57457ca0cc6769dab3caa9d1771ed6567a3
May 25 11:21:27.814410 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:27.814368    2576 generic.go:358] "Generic (PLEG): container finished" podID="a8391b8d-7b7f-4387-864d-63297d47d26e" containerID="9a1b53b74d7736e952c8554477abf397467bb468cabc0cad9215fc9c2e2752c4" exitCode=0
May 25 11:21:27.814410 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:27.814421    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a8391b8d-7b7f-4387-864d-63297d47d26e","Type":"ContainerDied","Data":"9a1b53b74d7736e952c8554477abf397467bb468cabc0cad9215fc9c2e2752c4"}
May 25 11:21:27.814651 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:27.814446    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a8391b8d-7b7f-4387-864d-63297d47d26e","Type":"ContainerStarted","Data":"62668ab23acb3fed67d0673c020ed57457ca0cc6769dab3caa9d1771ed6567a3"}
May 25 11:21:28.820145 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:28.820108    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a8391b8d-7b7f-4387-864d-63297d47d26e","Type":"ContainerStarted","Data":"c599c3b9c7f060bdf036b684d391537426c11d88ae4ea0279e6d34316b2d1bbf"}
May 25 11:21:28.820145 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:28.820147    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a8391b8d-7b7f-4387-864d-63297d47d26e","Type":"ContainerStarted","Data":"34a317bbede270c566765eef2a65fd02675e8085d8fb15fc75ac055bb5161ae3"}
May 25 11:21:28.820545 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:28.820157    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a8391b8d-7b7f-4387-864d-63297d47d26e","Type":"ContainerStarted","Data":"44f6c85347cbcc806d7745fdac113e3cc80d007e9504ada564679b9a31b54f2b"}
May 25 11:21:28.820545 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:28.820166    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a8391b8d-7b7f-4387-864d-63297d47d26e","Type":"ContainerStarted","Data":"bc293131b620bb31dd91fc4cd6ea7fe2916a8a0640ae2603fc3e0647e3affd09"}
May 25 11:21:28.820545 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:28.820174    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a8391b8d-7b7f-4387-864d-63297d47d26e","Type":"ContainerStarted","Data":"f4e353bb25fc57b395ce89b6098fd50c55aa9db4499fae66da84066174dfeb70"}
May 25 11:21:28.820545 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:28.820182    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a8391b8d-7b7f-4387-864d-63297d47d26e","Type":"ContainerStarted","Data":"1432cab1f681f5fb7b2e3db6b7645b96166fe8e5c8f14f6641a9306b93c9e22c"}
May 25 11:21:28.849736 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:28.849698    2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.849683938 podStartE2EDuration="2.849683938s" podCreationTimestamp="2026-05-25 11:21:26 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 11:21:28.849138848 +0000 UTC m=+234.366870304" watchObservedRunningTime="2026-05-25 11:21:28.849683938 +0000 UTC m=+234.367415395"
May 25 11:21:29.062969 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:29.062930    2576 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
May 25 11:21:29.063543 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:29.063402    2576 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="dced9206-c2e9-4a0d-9a21-4e5c898dcfa8" containerName="prometheus" containerID="cri-o://778d1fbfdd0cc431dc424cd5652fbade69c009d542c55ef45ebd318d04da8be4" gracePeriod=600
May 25 11:21:29.063543 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:29.063423    2576 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="dced9206-c2e9-4a0d-9a21-4e5c898dcfa8" containerName="kube-rbac-proxy" containerID="cri-o://88604427b05b40cf1e75d8da69d674c340583d68f6d0cc780f3544a10f20a8db" gracePeriod=600
May 25 11:21:29.063543 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:29.063455    2576 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="dced9206-c2e9-4a0d-9a21-4e5c898dcfa8" containerName="config-reloader" containerID="cri-o://847b784e0a2b0c6885cc1bbfc1ec0def50bde7f32ffa03e3faae57fae770585e" gracePeriod=600
May 25 11:21:29.063543 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:29.063475    2576 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="dced9206-c2e9-4a0d-9a21-4e5c898dcfa8" containerName="kube-rbac-proxy-thanos" containerID="cri-o://12cfb840a9340b7039ecb9d73b1e0d48b8ada060d53ea1591074812b161843f8" gracePeriod=600
May 25 11:21:29.063543 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:29.063434    2576 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="dced9206-c2e9-4a0d-9a21-4e5c898dcfa8" containerName="kube-rbac-proxy-web" containerID="cri-o://4a672a4b0c61e55e51c4947a70792e48e888143bb9743f0e1ddcd00ea2bac6e2" gracePeriod=600
May 25 11:21:29.063543 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:29.063414    2576 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="dced9206-c2e9-4a0d-9a21-4e5c898dcfa8" containerName="thanos-sidecar" containerID="cri-o://87f910240b3a5616eadc7d66e802e3e220ba4490cb29fb88164ac6cb97be4f78" gracePeriod=600
May 25 11:21:29.825957 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:29.825918    2576 generic.go:358] "Generic (PLEG): container finished" podID="dced9206-c2e9-4a0d-9a21-4e5c898dcfa8" containerID="12cfb840a9340b7039ecb9d73b1e0d48b8ada060d53ea1591074812b161843f8" exitCode=0
May 25 11:21:29.825957 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:29.825945    2576 generic.go:358] "Generic (PLEG): container finished" podID="dced9206-c2e9-4a0d-9a21-4e5c898dcfa8" containerID="88604427b05b40cf1e75d8da69d674c340583d68f6d0cc780f3544a10f20a8db" exitCode=0
May 25 11:21:29.825957 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:29.825954    2576 generic.go:358] "Generic (PLEG): container finished" podID="dced9206-c2e9-4a0d-9a21-4e5c898dcfa8" containerID="87f910240b3a5616eadc7d66e802e3e220ba4490cb29fb88164ac6cb97be4f78" exitCode=0
May 25 11:21:29.825957 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:29.825960    2576 generic.go:358] "Generic (PLEG): container finished" podID="dced9206-c2e9-4a0d-9a21-4e5c898dcfa8" containerID="847b784e0a2b0c6885cc1bbfc1ec0def50bde7f32ffa03e3faae57fae770585e" exitCode=0
May 25 11:21:29.825957 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:29.825965    2576 generic.go:358] "Generic (PLEG): container finished" podID="dced9206-c2e9-4a0d-9a21-4e5c898dcfa8" containerID="778d1fbfdd0cc431dc424cd5652fbade69c009d542c55ef45ebd318d04da8be4" exitCode=0
May 25 11:21:29.826467 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:29.825992    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"dced9206-c2e9-4a0d-9a21-4e5c898dcfa8","Type":"ContainerDied","Data":"12cfb840a9340b7039ecb9d73b1e0d48b8ada060d53ea1591074812b161843f8"}
May 25 11:21:29.826467 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:29.826024    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"dced9206-c2e9-4a0d-9a21-4e5c898dcfa8","Type":"ContainerDied","Data":"88604427b05b40cf1e75d8da69d674c340583d68f6d0cc780f3544a10f20a8db"}
May 25 11:21:29.826467 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:29.826034    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"dced9206-c2e9-4a0d-9a21-4e5c898dcfa8","Type":"ContainerDied","Data":"87f910240b3a5616eadc7d66e802e3e220ba4490cb29fb88164ac6cb97be4f78"}
May 25 11:21:29.826467 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:29.826043    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"dced9206-c2e9-4a0d-9a21-4e5c898dcfa8","Type":"ContainerDied","Data":"847b784e0a2b0c6885cc1bbfc1ec0def50bde7f32ffa03e3faae57fae770585e"}
May 25 11:21:29.826467 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:29.826051    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"dced9206-c2e9-4a0d-9a21-4e5c898dcfa8","Type":"ContainerDied","Data":"778d1fbfdd0cc431dc424cd5652fbade69c009d542c55ef45ebd318d04da8be4"}
May 25 11:21:30.317652 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:30.317631    2576 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
May 25 11:21:30.347222 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:30.347197    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dced9206-c2e9-4a0d-9a21-4e5c898dcfa8-prometheus-trusted-ca-bundle\") pod \"dced9206-c2e9-4a0d-9a21-4e5c898dcfa8\" (UID: \"dced9206-c2e9-4a0d-9a21-4e5c898dcfa8\") "
May 25 11:21:30.347385 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:30.347238    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/dced9206-c2e9-4a0d-9a21-4e5c898dcfa8-prometheus-k8s-db\") pod \"dced9206-c2e9-4a0d-9a21-4e5c898dcfa8\" (UID: \"dced9206-c2e9-4a0d-9a21-4e5c898dcfa8\") "
May 25 11:21:30.347385 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:30.347265    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dced9206-c2e9-4a0d-9a21-4e5c898dcfa8-configmap-serving-certs-ca-bundle\") pod \"dced9206-c2e9-4a0d-9a21-4e5c898dcfa8\" (UID: \"dced9206-c2e9-4a0d-9a21-4e5c898dcfa8\") "
May 25 11:21:30.347385 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:30.347318    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/dced9206-c2e9-4a0d-9a21-4e5c898dcfa8-secret-kube-rbac-proxy\") pod \"dced9206-c2e9-4a0d-9a21-4e5c898dcfa8\" (UID: \"dced9206-c2e9-4a0d-9a21-4e5c898dcfa8\") "
May 25 11:21:30.347385 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:30.347344    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/dced9206-c2e9-4a0d-9a21-4e5c898dcfa8-prometheus-k8s-rulefiles-0\") pod \"dced9206-c2e9-4a0d-9a21-4e5c898dcfa8\" (UID: \"dced9206-c2e9-4a0d-9a21-4e5c898dcfa8\") "
May 25 11:21:30.347385 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:30.347369    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8f55h\" (UniqueName: \"kubernetes.io/projected/dced9206-c2e9-4a0d-9a21-4e5c898dcfa8-kube-api-access-8f55h\") pod \"dced9206-c2e9-4a0d-9a21-4e5c898dcfa8\" (UID: \"dced9206-c2e9-4a0d-9a21-4e5c898dcfa8\") "
May 25 11:21:30.347742 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:30.347438    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/dced9206-c2e9-4a0d-9a21-4e5c898dcfa8-secret-grpc-tls\") pod \"dced9206-c2e9-4a0d-9a21-4e5c898dcfa8\" (UID: \"dced9206-c2e9-4a0d-9a21-4e5c898dcfa8\") "
May 25 11:21:30.347742 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:30.347471    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/dced9206-c2e9-4a0d-9a21-4e5c898dcfa8-config-out\") pod \"dced9206-c2e9-4a0d-9a21-4e5c898dcfa8\" (UID: \"dced9206-c2e9-4a0d-9a21-4e5c898dcfa8\") "
May 25 11:21:30.347742 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:30.347518    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dced9206-c2e9-4a0d-9a21-4e5c898dcfa8-configmap-kubelet-serving-ca-bundle\") pod \"dced9206-c2e9-4a0d-9a21-4e5c898dcfa8\" (UID: \"dced9206-c2e9-4a0d-9a21-4e5c898dcfa8\") "
May 25 11:21:30.347742 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:30.347560    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/dced9206-c2e9-4a0d-9a21-4e5c898dcfa8-secret-metrics-client-certs\") pod \"dced9206-c2e9-4a0d-9a21-4e5c898dcfa8\" (UID: \"dced9206-c2e9-4a0d-9a21-4e5c898dcfa8\") "
May 25 11:21:30.347742 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:30.347586    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/dced9206-c2e9-4a0d-9a21-4e5c898dcfa8-thanos-prometheus-http-client-file\") pod \"dced9206-c2e9-4a0d-9a21-4e5c898dcfa8\" (UID: \"dced9206-c2e9-4a0d-9a21-4e5c898dcfa8\") "
May 25 11:21:30.347742 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:30.347634    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dced9206-c2e9-4a0d-9a21-4e5c898dcfa8-configmap-metrics-client-ca\") pod \"dced9206-c2e9-4a0d-9a21-4e5c898dcfa8\" (UID: \"dced9206-c2e9-4a0d-9a21-4e5c898dcfa8\") "
May 25 11:21:30.347742 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:30.347655    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dced9206-c2e9-4a0d-9a21-4e5c898dcfa8-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "dced9206-c2e9-4a0d-9a21-4e5c898dcfa8" (UID: "dced9206-c2e9-4a0d-9a21-4e5c898dcfa8"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 11:21:30.347742 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:30.347660    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/dced9206-c2e9-4a0d-9a21-4e5c898dcfa8-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"dced9206-c2e9-4a0d-9a21-4e5c898dcfa8\" (UID: \"dced9206-c2e9-4a0d-9a21-4e5c898dcfa8\") "
May 25 11:21:30.347742 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:30.347720    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/dced9206-c2e9-4a0d-9a21-4e5c898dcfa8-secret-prometheus-k8s-tls\") pod \"dced9206-c2e9-4a0d-9a21-4e5c898dcfa8\" (UID: \"dced9206-c2e9-4a0d-9a21-4e5c898dcfa8\") "
May 25 11:21:30.348144 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:30.347754    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/dced9206-c2e9-4a0d-9a21-4e5c898dcfa8-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"dced9206-c2e9-4a0d-9a21-4e5c898dcfa8\" (UID: \"dced9206-c2e9-4a0d-9a21-4e5c898dcfa8\") "
May 25 11:21:30.348144 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:30.347800    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/dced9206-c2e9-4a0d-9a21-4e5c898dcfa8-config\") pod \"dced9206-c2e9-4a0d-9a21-4e5c898dcfa8\" (UID: \"dced9206-c2e9-4a0d-9a21-4e5c898dcfa8\") "
May 25 11:21:30.348144 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:30.347835    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/dced9206-c2e9-4a0d-9a21-4e5c898dcfa8-web-config\") pod \"dced9206-c2e9-4a0d-9a21-4e5c898dcfa8\" (UID: \"dced9206-c2e9-4a0d-9a21-4e5c898dcfa8\") "
May 25 11:21:30.348144 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:30.347859    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/dced9206-c2e9-4a0d-9a21-4e5c898dcfa8-tls-assets\") pod \"dced9206-c2e9-4a0d-9a21-4e5c898dcfa8\" (UID: \"dced9206-c2e9-4a0d-9a21-4e5c898dcfa8\") "
May 25 11:21:30.348144 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:30.348127    2576 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dced9206-c2e9-4a0d-9a21-4e5c898dcfa8-prometheus-trusted-ca-bundle\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
May 25 11:21:30.348699 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:30.348464    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dced9206-c2e9-4a0d-9a21-4e5c898dcfa8-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "dced9206-c2e9-4a0d-9a21-4e5c898dcfa8" (UID: "dced9206-c2e9-4a0d-9a21-4e5c898dcfa8"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:21:30.348699 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:30.348450    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dced9206-c2e9-4a0d-9a21-4e5c898dcfa8-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "dced9206-c2e9-4a0d-9a21-4e5c898dcfa8" (UID: "dced9206-c2e9-4a0d-9a21-4e5c898dcfa8"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 11:21:30.348941 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:30.348914    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dced9206-c2e9-4a0d-9a21-4e5c898dcfa8-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "dced9206-c2e9-4a0d-9a21-4e5c898dcfa8" (UID: "dced9206-c2e9-4a0d-9a21-4e5c898dcfa8"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 11:21:30.350066 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:30.350035    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dced9206-c2e9-4a0d-9a21-4e5c898dcfa8-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "dced9206-c2e9-4a0d-9a21-4e5c898dcfa8" (UID: "dced9206-c2e9-4a0d-9a21-4e5c898dcfa8"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 11:21:30.350648 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:30.350619    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dced9206-c2e9-4a0d-9a21-4e5c898dcfa8-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "dced9206-c2e9-4a0d-9a21-4e5c898dcfa8" (UID: "dced9206-c2e9-4a0d-9a21-4e5c898dcfa8"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 11:21:30.351190 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:30.351152    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dced9206-c2e9-4a0d-9a21-4e5c898dcfa8-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "dced9206-c2e9-4a0d-9a21-4e5c898dcfa8" (UID: "dced9206-c2e9-4a0d-9a21-4e5c898dcfa8"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 11:21:30.351472 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:30.351441    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dced9206-c2e9-4a0d-9a21-4e5c898dcfa8-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "dced9206-c2e9-4a0d-9a21-4e5c898dcfa8" (UID: "dced9206-c2e9-4a0d-9a21-4e5c898dcfa8"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 11:21:30.353015 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:30.352991    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dced9206-c2e9-4a0d-9a21-4e5c898dcfa8-config" (OuterVolumeSpecName: "config") pod "dced9206-c2e9-4a0d-9a21-4e5c898dcfa8" (UID: "dced9206-c2e9-4a0d-9a21-4e5c898dcfa8"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 11:21:30.353107 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:30.353011    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dced9206-c2e9-4a0d-9a21-4e5c898dcfa8-kube-api-access-8f55h" (OuterVolumeSpecName: "kube-api-access-8f55h") pod "dced9206-c2e9-4a0d-9a21-4e5c898dcfa8" (UID: "dced9206-c2e9-4a0d-9a21-4e5c898dcfa8"). InnerVolumeSpecName "kube-api-access-8f55h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 11:21:30.353169 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:30.353111    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dced9206-c2e9-4a0d-9a21-4e5c898dcfa8-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "dced9206-c2e9-4a0d-9a21-4e5c898dcfa8" (UID: "dced9206-c2e9-4a0d-9a21-4e5c898dcfa8"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 11:21:30.353468 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:30.353423    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dced9206-c2e9-4a0d-9a21-4e5c898dcfa8-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "dced9206-c2e9-4a0d-9a21-4e5c898dcfa8" (UID: "dced9206-c2e9-4a0d-9a21-4e5c898dcfa8"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 11:21:30.353468 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:30.353437    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dced9206-c2e9-4a0d-9a21-4e5c898dcfa8-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "dced9206-c2e9-4a0d-9a21-4e5c898dcfa8" (UID: "dced9206-c2e9-4a0d-9a21-4e5c898dcfa8"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 11:21:30.354897 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:30.354871    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dced9206-c2e9-4a0d-9a21-4e5c898dcfa8-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "dced9206-c2e9-4a0d-9a21-4e5c898dcfa8" (UID: "dced9206-c2e9-4a0d-9a21-4e5c898dcfa8"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 11:21:30.355561 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:30.355510    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dced9206-c2e9-4a0d-9a21-4e5c898dcfa8-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "dced9206-c2e9-4a0d-9a21-4e5c898dcfa8" (UID: "dced9206-c2e9-4a0d-9a21-4e5c898dcfa8"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 11:21:30.355561 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:30.355523    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dced9206-c2e9-4a0d-9a21-4e5c898dcfa8-config-out" (OuterVolumeSpecName: "config-out") pod "dced9206-c2e9-4a0d-9a21-4e5c898dcfa8" (UID: "dced9206-c2e9-4a0d-9a21-4e5c898dcfa8"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:21:30.356001 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:30.355968    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dced9206-c2e9-4a0d-9a21-4e5c898dcfa8-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "dced9206-c2e9-4a0d-9a21-4e5c898dcfa8" (UID: "dced9206-c2e9-4a0d-9a21-4e5c898dcfa8"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 11:21:30.365965 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:30.365910    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dced9206-c2e9-4a0d-9a21-4e5c898dcfa8-web-config" (OuterVolumeSpecName: "web-config") pod "dced9206-c2e9-4a0d-9a21-4e5c898dcfa8" (UID: "dced9206-c2e9-4a0d-9a21-4e5c898dcfa8"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 11:21:30.449032 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:30.448995    2576 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/dced9206-c2e9-4a0d-9a21-4e5c898dcfa8-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
May 25 11:21:30.449032 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:30.449030    2576 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/dced9206-c2e9-4a0d-9a21-4e5c898dcfa8-config\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
May 25 11:21:30.449032 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:30.449041    2576 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/dced9206-c2e9-4a0d-9a21-4e5c898dcfa8-web-config\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
May 25 11:21:30.449032 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:30.449050    2576 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/dced9206-c2e9-4a0d-9a21-4e5c898dcfa8-tls-assets\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
May 25 11:21:30.449261 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:30.449058    2576 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/dced9206-c2e9-4a0d-9a21-4e5c898dcfa8-prometheus-k8s-db\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
May 25 11:21:30.449261 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:30.449066    2576 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dced9206-c2e9-4a0d-9a21-4e5c898dcfa8-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
May 25 11:21:30.449261 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:30.449075    2576 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/dced9206-c2e9-4a0d-9a21-4e5c898dcfa8-secret-kube-rbac-proxy\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
May 25 11:21:30.449261 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:30.449083    2576 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/dced9206-c2e9-4a0d-9a21-4e5c898dcfa8-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
May 25 11:21:30.449261 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:30.449091    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8f55h\" (UniqueName: \"kubernetes.io/projected/dced9206-c2e9-4a0d-9a21-4e5c898dcfa8-kube-api-access-8f55h\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
May 25 11:21:30.449261 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:30.449103    2576 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/dced9206-c2e9-4a0d-9a21-4e5c898dcfa8-secret-grpc-tls\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
May 25 11:21:30.449261 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:30.449112    2576 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/dced9206-c2e9-4a0d-9a21-4e5c898dcfa8-config-out\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
May 25 11:21:30.449261 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:30.449120    2576 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dced9206-c2e9-4a0d-9a21-4e5c898dcfa8-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
May 25 11:21:30.449261 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:30.449128    2576 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/dced9206-c2e9-4a0d-9a21-4e5c898dcfa8-secret-metrics-client-certs\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
May 25 11:21:30.449261 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:30.449138    2576 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/dced9206-c2e9-4a0d-9a21-4e5c898dcfa8-thanos-prometheus-http-client-file\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
May 25 11:21:30.449261 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:30.449146    2576 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dced9206-c2e9-4a0d-9a21-4e5c898dcfa8-configmap-metrics-client-ca\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
May 25 11:21:30.449261 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:30.449155    2576 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/dced9206-c2e9-4a0d-9a21-4e5c898dcfa8-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
May 25 11:21:30.449261 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:30.449164    2576 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/dced9206-c2e9-4a0d-9a21-4e5c898dcfa8-secret-prometheus-k8s-tls\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
May 25 11:21:30.832087 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:30.832053    2576 generic.go:358] "Generic (PLEG): container finished" podID="dced9206-c2e9-4a0d-9a21-4e5c898dcfa8" containerID="4a672a4b0c61e55e51c4947a70792e48e888143bb9743f0e1ddcd00ea2bac6e2" exitCode=0
May 25 11:21:30.832515 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:30.832184    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"dced9206-c2e9-4a0d-9a21-4e5c898dcfa8","Type":"ContainerDied","Data":"4a672a4b0c61e55e51c4947a70792e48e888143bb9743f0e1ddcd00ea2bac6e2"}
May 25 11:21:30.832515 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:30.832216    2576 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
May 25 11:21:30.832515 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:30.832233    2576 scope.go:117] "RemoveContainer" containerID="12cfb840a9340b7039ecb9d73b1e0d48b8ada060d53ea1591074812b161843f8"
May 25 11:21:30.832515 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:30.832221    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"dced9206-c2e9-4a0d-9a21-4e5c898dcfa8","Type":"ContainerDied","Data":"de9b23ff4aa2a941394f49dbc033d3ce0d211570bbb5c5aaa262b42c65a57458"}
May 25 11:21:30.839684 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:30.839665    2576 scope.go:117] "RemoveContainer" containerID="88604427b05b40cf1e75d8da69d674c340583d68f6d0cc780f3544a10f20a8db"
May 25 11:21:30.846504 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:30.846486    2576 scope.go:117] "RemoveContainer" containerID="4a672a4b0c61e55e51c4947a70792e48e888143bb9743f0e1ddcd00ea2bac6e2"
May 25 11:21:30.852581 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:30.852561    2576 scope.go:117] "RemoveContainer" containerID="87f910240b3a5616eadc7d66e802e3e220ba4490cb29fb88164ac6cb97be4f78"
May 25 11:21:30.858829 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:30.858804    2576 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
May 25 11:21:30.859073 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:30.859059    2576 scope.go:117] "RemoveContainer" containerID="847b784e0a2b0c6885cc1bbfc1ec0def50bde7f32ffa03e3faae57fae770585e"
May 25 11:21:30.862572 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:30.862553    2576 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
May 25 11:21:30.865895 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:30.865873    2576 scope.go:117] "RemoveContainer" containerID="778d1fbfdd0cc431dc424cd5652fbade69c009d542c55ef45ebd318d04da8be4"
May 25 11:21:30.872469 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:30.872453    2576 scope.go:117] "RemoveContainer" containerID="6230df5b3509ac11cd9e912e5eb0306e5c493e1f86b8fa12e6aca56a4d789af9"
May 25 11:21:30.880337 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:30.880313    2576 scope.go:117] "RemoveContainer" containerID="12cfb840a9340b7039ecb9d73b1e0d48b8ada060d53ea1591074812b161843f8"
May 25 11:21:30.880597 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:21:30.880580    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12cfb840a9340b7039ecb9d73b1e0d48b8ada060d53ea1591074812b161843f8\": container with ID starting with 12cfb840a9340b7039ecb9d73b1e0d48b8ada060d53ea1591074812b161843f8 not found: ID does not exist" containerID="12cfb840a9340b7039ecb9d73b1e0d48b8ada060d53ea1591074812b161843f8"
May 25 11:21:30.880658 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:30.880604    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12cfb840a9340b7039ecb9d73b1e0d48b8ada060d53ea1591074812b161843f8"} err="failed to get container status \"12cfb840a9340b7039ecb9d73b1e0d48b8ada060d53ea1591074812b161843f8\": rpc error: code = NotFound desc = could not find container \"12cfb840a9340b7039ecb9d73b1e0d48b8ada060d53ea1591074812b161843f8\": container with ID starting with 12cfb840a9340b7039ecb9d73b1e0d48b8ada060d53ea1591074812b161843f8 not found: ID does not exist"
May 25 11:21:30.880658 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:30.880623    2576 scope.go:117] "RemoveContainer" containerID="88604427b05b40cf1e75d8da69d674c340583d68f6d0cc780f3544a10f20a8db"
May 25 11:21:30.880884 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:21:30.880863    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88604427b05b40cf1e75d8da69d674c340583d68f6d0cc780f3544a10f20a8db\": container with ID starting with 88604427b05b40cf1e75d8da69d674c340583d68f6d0cc780f3544a10f20a8db not found: ID does not exist" containerID="88604427b05b40cf1e75d8da69d674c340583d68f6d0cc780f3544a10f20a8db"
May 25 11:21:30.880927 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:30.880888    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88604427b05b40cf1e75d8da69d674c340583d68f6d0cc780f3544a10f20a8db"} err="failed to get container status \"88604427b05b40cf1e75d8da69d674c340583d68f6d0cc780f3544a10f20a8db\": rpc error: code = NotFound desc = could not find container \"88604427b05b40cf1e75d8da69d674c340583d68f6d0cc780f3544a10f20a8db\": container with ID starting with 88604427b05b40cf1e75d8da69d674c340583d68f6d0cc780f3544a10f20a8db not found: ID does not exist"
May 25 11:21:30.880927 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:30.880907    2576 scope.go:117] "RemoveContainer" containerID="4a672a4b0c61e55e51c4947a70792e48e888143bb9743f0e1ddcd00ea2bac6e2"
May 25 11:21:30.881150 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:21:30.881133    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a672a4b0c61e55e51c4947a70792e48e888143bb9743f0e1ddcd00ea2bac6e2\": container with ID starting with 4a672a4b0c61e55e51c4947a70792e48e888143bb9743f0e1ddcd00ea2bac6e2 not found: ID does not exist" containerID="4a672a4b0c61e55e51c4947a70792e48e888143bb9743f0e1ddcd00ea2bac6e2"
May 25 11:21:30.881194 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:30.881154    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a672a4b0c61e55e51c4947a70792e48e888143bb9743f0e1ddcd00ea2bac6e2"} err="failed to get container status \"4a672a4b0c61e55e51c4947a70792e48e888143bb9743f0e1ddcd00ea2bac6e2\": rpc error: code = NotFound desc = could not find container \"4a672a4b0c61e55e51c4947a70792e48e888143bb9743f0e1ddcd00ea2bac6e2\": container with ID starting with 4a672a4b0c61e55e51c4947a70792e48e888143bb9743f0e1ddcd00ea2bac6e2 not found: ID does not exist"
May 25 11:21:30.881194 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:30.881167    2576 scope.go:117] "RemoveContainer" containerID="87f910240b3a5616eadc7d66e802e3e220ba4490cb29fb88164ac6cb97be4f78"
May 25 11:21:30.881460 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:21:30.881445    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87f910240b3a5616eadc7d66e802e3e220ba4490cb29fb88164ac6cb97be4f78\": container with ID starting with 87f910240b3a5616eadc7d66e802e3e220ba4490cb29fb88164ac6cb97be4f78 not found: ID does not exist" containerID="87f910240b3a5616eadc7d66e802e3e220ba4490cb29fb88164ac6cb97be4f78"
May 25 11:21:30.881503 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:30.881463    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87f910240b3a5616eadc7d66e802e3e220ba4490cb29fb88164ac6cb97be4f78"} err="failed to get container status \"87f910240b3a5616eadc7d66e802e3e220ba4490cb29fb88164ac6cb97be4f78\": rpc error: code = NotFound desc = could not find container \"87f910240b3a5616eadc7d66e802e3e220ba4490cb29fb88164ac6cb97be4f78\": container with ID starting with 87f910240b3a5616eadc7d66e802e3e220ba4490cb29fb88164ac6cb97be4f78 not found: ID does not exist"
May 25 11:21:30.881503 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:30.881485    2576 scope.go:117] "RemoveContainer" containerID="847b784e0a2b0c6885cc1bbfc1ec0def50bde7f32ffa03e3faae57fae770585e"
May 25 11:21:30.881715 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:21:30.881697    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"847b784e0a2b0c6885cc1bbfc1ec0def50bde7f32ffa03e3faae57fae770585e\": container with ID starting with 847b784e0a2b0c6885cc1bbfc1ec0def50bde7f32ffa03e3faae57fae770585e not found: ID does not exist" containerID="847b784e0a2b0c6885cc1bbfc1ec0def50bde7f32ffa03e3faae57fae770585e"
May 25 11:21:30.881766 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:30.881718    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"847b784e0a2b0c6885cc1bbfc1ec0def50bde7f32ffa03e3faae57fae770585e"} err="failed to get container status \"847b784e0a2b0c6885cc1bbfc1ec0def50bde7f32ffa03e3faae57fae770585e\": rpc error: code = NotFound desc = could not find container \"847b784e0a2b0c6885cc1bbfc1ec0def50bde7f32ffa03e3faae57fae770585e\": container with ID starting with 847b784e0a2b0c6885cc1bbfc1ec0def50bde7f32ffa03e3faae57fae770585e not found: ID does not exist"
May 25 11:21:30.881766 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:30.881742    2576 scope.go:117] "RemoveContainer" containerID="778d1fbfdd0cc431dc424cd5652fbade69c009d542c55ef45ebd318d04da8be4"
May 25 11:21:30.881970 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:21:30.881955    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"778d1fbfdd0cc431dc424cd5652fbade69c009d542c55ef45ebd318d04da8be4\": container with ID starting with 778d1fbfdd0cc431dc424cd5652fbade69c009d542c55ef45ebd318d04da8be4 not found: ID does not exist" containerID="778d1fbfdd0cc431dc424cd5652fbade69c009d542c55ef45ebd318d04da8be4"
May 25 11:21:30.882010 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:30.881974    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"778d1fbfdd0cc431dc424cd5652fbade69c009d542c55ef45ebd318d04da8be4"} err="failed to get container status \"778d1fbfdd0cc431dc424cd5652fbade69c009d542c55ef45ebd318d04da8be4\": rpc error: code = NotFound desc = could not find container \"778d1fbfdd0cc431dc424cd5652fbade69c009d542c55ef45ebd318d04da8be4\": container with ID starting with 778d1fbfdd0cc431dc424cd5652fbade69c009d542c55ef45ebd318d04da8be4 not found: ID does not exist"
May 25 11:21:30.882010 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:30.881986    2576 scope.go:117] "RemoveContainer" containerID="6230df5b3509ac11cd9e912e5eb0306e5c493e1f86b8fa12e6aca56a4d789af9"
May 25 11:21:30.882234 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:21:30.882197    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6230df5b3509ac11cd9e912e5eb0306e5c493e1f86b8fa12e6aca56a4d789af9\": container with ID starting with 6230df5b3509ac11cd9e912e5eb0306e5c493e1f86b8fa12e6aca56a4d789af9 not found: ID does not exist" containerID="6230df5b3509ac11cd9e912e5eb0306e5c493e1f86b8fa12e6aca56a4d789af9"
May 25 11:21:30.882303 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:30.882236    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6230df5b3509ac11cd9e912e5eb0306e5c493e1f86b8fa12e6aca56a4d789af9"} err="failed to get container status \"6230df5b3509ac11cd9e912e5eb0306e5c493e1f86b8fa12e6aca56a4d789af9\": rpc error: code = NotFound desc = could not find container \"6230df5b3509ac11cd9e912e5eb0306e5c493e1f86b8fa12e6aca56a4d789af9\": container with ID starting with 6230df5b3509ac11cd9e912e5eb0306e5c493e1f86b8fa12e6aca56a4d789af9 not found: ID does not exist"
May 25 11:21:30.888067 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:30.888049    2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
May 25 11:21:30.888342 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:30.888330    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dced9206-c2e9-4a0d-9a21-4e5c898dcfa8" containerName="prometheus"
May 25 11:21:30.888389 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:30.888344    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="dced9206-c2e9-4a0d-9a21-4e5c898dcfa8" containerName="prometheus"
May 25 11:21:30.888389 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:30.888356    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dced9206-c2e9-4a0d-9a21-4e5c898dcfa8" containerName="thanos-sidecar"
May 25 11:21:30.888389 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:30.888361    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="dced9206-c2e9-4a0d-9a21-4e5c898dcfa8" containerName="thanos-sidecar"
May 25 11:21:30.888389 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:30.888368    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dced9206-c2e9-4a0d-9a21-4e5c898dcfa8" containerName="kube-rbac-proxy"
May 25 11:21:30.888389 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:30.888374    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="dced9206-c2e9-4a0d-9a21-4e5c898dcfa8" containerName="kube-rbac-proxy"
May 25 11:21:30.888389 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:30.888380    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dced9206-c2e9-4a0d-9a21-4e5c898dcfa8" containerName="kube-rbac-proxy-thanos"
May 25 11:21:30.888389 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:30.888385    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="dced9206-c2e9-4a0d-9a21-4e5c898dcfa8" containerName="kube-rbac-proxy-thanos"
May 25 11:21:30.888587 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:30.888396    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dced9206-c2e9-4a0d-9a21-4e5c898dcfa8" containerName="init-config-reloader"
May 25 11:21:30.888587 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:30.888401    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="dced9206-c2e9-4a0d-9a21-4e5c898dcfa8" containerName="init-config-reloader"
May 25 11:21:30.888587 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:30.888411    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dced9206-c2e9-4a0d-9a21-4e5c898dcfa8" containerName="kube-rbac-proxy-web"
May 25 11:21:30.888587 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:30.888417    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="dced9206-c2e9-4a0d-9a21-4e5c898dcfa8" containerName="kube-rbac-proxy-web"
May 25 11:21:30.888587 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:30.888424    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dced9206-c2e9-4a0d-9a21-4e5c898dcfa8" containerName="config-reloader"
May 25 11:21:30.888587 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:30.888428    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="dced9206-c2e9-4a0d-9a21-4e5c898dcfa8" containerName="config-reloader"
May 25 11:21:30.888587 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:30.888468    2576 memory_manager.go:356] "RemoveStaleState removing state" podUID="dced9206-c2e9-4a0d-9a21-4e5c898dcfa8" containerName="config-reloader"
May 25 11:21:30.888587 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:30.888478    2576 memory_manager.go:356] "RemoveStaleState removing state" podUID="dced9206-c2e9-4a0d-9a21-4e5c898dcfa8" containerName="kube-rbac-proxy"
May 25 11:21:30.888587 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:30.888486    2576 memory_manager.go:356] "RemoveStaleState removing state" podUID="dced9206-c2e9-4a0d-9a21-4e5c898dcfa8" containerName="kube-rbac-proxy-web"
May 25 11:21:30.888587 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:30.888493    2576 memory_manager.go:356] "RemoveStaleState removing state" podUID="dced9206-c2e9-4a0d-9a21-4e5c898dcfa8" containerName="prometheus"
May 25 11:21:30.888587 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:30.888499    2576 memory_manager.go:356] "RemoveStaleState removing state" podUID="dced9206-c2e9-4a0d-9a21-4e5c898dcfa8" containerName="kube-rbac-proxy-thanos"
May 25 11:21:30.888587 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:30.888505    2576 memory_manager.go:356] "RemoveStaleState removing state" podUID="dced9206-c2e9-4a0d-9a21-4e5c898dcfa8" containerName="thanos-sidecar"
May 25 11:21:30.893779 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:30.893764    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
May 25 11:21:30.895902 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:30.895882    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
May 25 11:21:30.896009 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:30.895918    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
May 25 11:21:30.896009 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:30.895930    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
May 25 11:21:30.896009 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:30.895957    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-nvblc\""
May 25 11:21:30.896168 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:30.896039    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
May 25 11:21:30.896305 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:30.896270    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
May 25 11:21:30.896475 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:30.896458    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
May 25 11:21:30.896628 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:30.896617    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
May 25 11:21:30.896896 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:30.896880    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
May 25 11:21:30.896954 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:30.896940    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-bobbetstr1scc\""
May 25 11:21:30.897593 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:30.897573    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
May 25 11:21:30.897593 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:30.897581    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
May 25 11:21:30.899378 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:30.899354    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
May 25 11:21:30.903605 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:30.903586    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
May 25 11:21:30.905590 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:30.905570    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
May 25 11:21:30.953456 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:30.953417    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/bb1f4e62-0113-40c8-b837-8f0a75d4f2c9-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"bb1f4e62-0113-40c8-b837-8f0a75d4f2c9\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 11:21:30.953456 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:30.953459    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bb1f4e62-0113-40c8-b837-8f0a75d4f2c9-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bb1f4e62-0113-40c8-b837-8f0a75d4f2c9\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 11:21:30.953651 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:30.953521    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/bb1f4e62-0113-40c8-b837-8f0a75d4f2c9-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"bb1f4e62-0113-40c8-b837-8f0a75d4f2c9\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 11:21:30.953651 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:30.953553    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/bb1f4e62-0113-40c8-b837-8f0a75d4f2c9-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"bb1f4e62-0113-40c8-b837-8f0a75d4f2c9\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 11:21:30.953651 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:30.953598    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bb1f4e62-0113-40c8-b837-8f0a75d4f2c9-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"bb1f4e62-0113-40c8-b837-8f0a75d4f2c9\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 11:21:30.953651 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:30.953630    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/bb1f4e62-0113-40c8-b837-8f0a75d4f2c9-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"bb1f4e62-0113-40c8-b837-8f0a75d4f2c9\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 11:21:30.953860 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:30.953675    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/bb1f4e62-0113-40c8-b837-8f0a75d4f2c9-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"bb1f4e62-0113-40c8-b837-8f0a75d4f2c9\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 11:21:30.953860 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:30.953696    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/bb1f4e62-0113-40c8-b837-8f0a75d4f2c9-web-config\") pod \"prometheus-k8s-0\" (UID: \"bb1f4e62-0113-40c8-b837-8f0a75d4f2c9\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 11:21:30.953860 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:30.953723    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/bb1f4e62-0113-40c8-b837-8f0a75d4f2c9-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"bb1f4e62-0113-40c8-b837-8f0a75d4f2c9\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 11:21:30.953860 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:30.953739    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/bb1f4e62-0113-40c8-b837-8f0a75d4f2c9-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"bb1f4e62-0113-40c8-b837-8f0a75d4f2c9\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 11:21:30.953860 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:30.953765    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/bb1f4e62-0113-40c8-b837-8f0a75d4f2c9-config\") pod \"prometheus-k8s-0\" (UID: \"bb1f4e62-0113-40c8-b837-8f0a75d4f2c9\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 11:21:30.953860 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:30.953808    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bb1f4e62-0113-40c8-b837-8f0a75d4f2c9-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bb1f4e62-0113-40c8-b837-8f0a75d4f2c9\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 11:21:30.953860 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:30.953843    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7dqkl\" (UniqueName: \"kubernetes.io/projected/bb1f4e62-0113-40c8-b837-8f0a75d4f2c9-kube-api-access-7dqkl\") pod \"prometheus-k8s-0\" (UID: \"bb1f4e62-0113-40c8-b837-8f0a75d4f2c9\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 11:21:30.954122 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:30.953872    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/bb1f4e62-0113-40c8-b837-8f0a75d4f2c9-config-out\") pod \"prometheus-k8s-0\" (UID: \"bb1f4e62-0113-40c8-b837-8f0a75d4f2c9\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 11:21:30.954122 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:30.953901    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/bb1f4e62-0113-40c8-b837-8f0a75d4f2c9-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"bb1f4e62-0113-40c8-b837-8f0a75d4f2c9\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 11:21:30.954122 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:30.953928    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/bb1f4e62-0113-40c8-b837-8f0a75d4f2c9-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"bb1f4e62-0113-40c8-b837-8f0a75d4f2c9\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 11:21:30.954122 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:30.953956    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/bb1f4e62-0113-40c8-b837-8f0a75d4f2c9-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"bb1f4e62-0113-40c8-b837-8f0a75d4f2c9\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 11:21:30.954122 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:30.953988    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bb1f4e62-0113-40c8-b837-8f0a75d4f2c9-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bb1f4e62-0113-40c8-b837-8f0a75d4f2c9\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 11:21:31.055335 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:31.055300    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/bb1f4e62-0113-40c8-b837-8f0a75d4f2c9-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"bb1f4e62-0113-40c8-b837-8f0a75d4f2c9\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 11:21:31.055512 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:31.055345    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/bb1f4e62-0113-40c8-b837-8f0a75d4f2c9-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"bb1f4e62-0113-40c8-b837-8f0a75d4f2c9\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 11:21:31.055512 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:31.055368    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bb1f4e62-0113-40c8-b837-8f0a75d4f2c9-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"bb1f4e62-0113-40c8-b837-8f0a75d4f2c9\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 11:21:31.055512 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:31.055387    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/bb1f4e62-0113-40c8-b837-8f0a75d4f2c9-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"bb1f4e62-0113-40c8-b837-8f0a75d4f2c9\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 11:21:31.055512 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:31.055430    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/bb1f4e62-0113-40c8-b837-8f0a75d4f2c9-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"bb1f4e62-0113-40c8-b837-8f0a75d4f2c9\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 11:21:31.055512 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:31.055447    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/bb1f4e62-0113-40c8-b837-8f0a75d4f2c9-web-config\") pod \"prometheus-k8s-0\" (UID: \"bb1f4e62-0113-40c8-b837-8f0a75d4f2c9\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 11:21:31.055512 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:31.055472    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/bb1f4e62-0113-40c8-b837-8f0a75d4f2c9-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"bb1f4e62-0113-40c8-b837-8f0a75d4f2c9\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 11:21:31.055512 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:31.055496    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/bb1f4e62-0113-40c8-b837-8f0a75d4f2c9-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"bb1f4e62-0113-40c8-b837-8f0a75d4f2c9\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 11:21:31.055839 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:31.055524    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/bb1f4e62-0113-40c8-b837-8f0a75d4f2c9-config\") pod \"prometheus-k8s-0\" (UID: \"bb1f4e62-0113-40c8-b837-8f0a75d4f2c9\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 11:21:31.055839 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:31.055549    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bb1f4e62-0113-40c8-b837-8f0a75d4f2c9-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bb1f4e62-0113-40c8-b837-8f0a75d4f2c9\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 11:21:31.055839 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:31.055578    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-7dqkl\" (UniqueName: \"kubernetes.io/projected/bb1f4e62-0113-40c8-b837-8f0a75d4f2c9-kube-api-access-7dqkl\") pod \"prometheus-k8s-0\" (UID: \"bb1f4e62-0113-40c8-b837-8f0a75d4f2c9\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 11:21:31.055839 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:31.055605    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/bb1f4e62-0113-40c8-b837-8f0a75d4f2c9-config-out\") pod \"prometheus-k8s-0\" (UID: \"bb1f4e62-0113-40c8-b837-8f0a75d4f2c9\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 11:21:31.055839 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:31.055632    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/bb1f4e62-0113-40c8-b837-8f0a75d4f2c9-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"bb1f4e62-0113-40c8-b837-8f0a75d4f2c9\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 11:21:31.055839 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:31.055655    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/bb1f4e62-0113-40c8-b837-8f0a75d4f2c9-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"bb1f4e62-0113-40c8-b837-8f0a75d4f2c9\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 11:21:31.055839 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:31.055682    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/bb1f4e62-0113-40c8-b837-8f0a75d4f2c9-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"bb1f4e62-0113-40c8-b837-8f0a75d4f2c9\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 11:21:31.055839 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:31.055715    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bb1f4e62-0113-40c8-b837-8f0a75d4f2c9-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bb1f4e62-0113-40c8-b837-8f0a75d4f2c9\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 11:21:31.055839 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:31.055768    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/bb1f4e62-0113-40c8-b837-8f0a75d4f2c9-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"bb1f4e62-0113-40c8-b837-8f0a75d4f2c9\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 11:21:31.055839 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:31.055792    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bb1f4e62-0113-40c8-b837-8f0a75d4f2c9-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bb1f4e62-0113-40c8-b837-8f0a75d4f2c9\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 11:21:31.056337 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:31.055851    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/bb1f4e62-0113-40c8-b837-8f0a75d4f2c9-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"bb1f4e62-0113-40c8-b837-8f0a75d4f2c9\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 11:21:31.056337 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:31.056210    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bb1f4e62-0113-40c8-b837-8f0a75d4f2c9-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"bb1f4e62-0113-40c8-b837-8f0a75d4f2c9\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 11:21:31.056687 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:31.056660    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bb1f4e62-0113-40c8-b837-8f0a75d4f2c9-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bb1f4e62-0113-40c8-b837-8f0a75d4f2c9\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 11:21:31.056769 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:31.056706    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bb1f4e62-0113-40c8-b837-8f0a75d4f2c9-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bb1f4e62-0113-40c8-b837-8f0a75d4f2c9\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 11:21:31.058826 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:31.058504    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/bb1f4e62-0113-40c8-b837-8f0a75d4f2c9-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"bb1f4e62-0113-40c8-b837-8f0a75d4f2c9\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 11:21:31.058826 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:31.058529    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/bb1f4e62-0113-40c8-b837-8f0a75d4f2c9-config-out\") pod \"prometheus-k8s-0\" (UID: \"bb1f4e62-0113-40c8-b837-8f0a75d4f2c9\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 11:21:31.058826 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:31.058541    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/bb1f4e62-0113-40c8-b837-8f0a75d4f2c9-web-config\") pod \"prometheus-k8s-0\" (UID: \"bb1f4e62-0113-40c8-b837-8f0a75d4f2c9\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 11:21:31.058826 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:31.058641    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/bb1f4e62-0113-40c8-b837-8f0a75d4f2c9-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"bb1f4e62-0113-40c8-b837-8f0a75d4f2c9\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 11:21:31.059156 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:31.059009    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/bb1f4e62-0113-40c8-b837-8f0a75d4f2c9-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"bb1f4e62-0113-40c8-b837-8f0a75d4f2c9\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 11:21:31.059456 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:31.059371    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bb1f4e62-0113-40c8-b837-8f0a75d4f2c9-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bb1f4e62-0113-40c8-b837-8f0a75d4f2c9\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 11:21:31.059456 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:31.059386    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/bb1f4e62-0113-40c8-b837-8f0a75d4f2c9-config\") pod \"prometheus-k8s-0\" (UID: \"bb1f4e62-0113-40c8-b837-8f0a75d4f2c9\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 11:21:31.059456 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:31.059457    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/bb1f4e62-0113-40c8-b837-8f0a75d4f2c9-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"bb1f4e62-0113-40c8-b837-8f0a75d4f2c9\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 11:21:31.059719 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:31.059701    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/bb1f4e62-0113-40c8-b837-8f0a75d4f2c9-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"bb1f4e62-0113-40c8-b837-8f0a75d4f2c9\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 11:21:31.061264 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:31.061239    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/bb1f4e62-0113-40c8-b837-8f0a75d4f2c9-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"bb1f4e62-0113-40c8-b837-8f0a75d4f2c9\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 11:21:31.061378 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:31.061355    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/bb1f4e62-0113-40c8-b837-8f0a75d4f2c9-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"bb1f4e62-0113-40c8-b837-8f0a75d4f2c9\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 11:21:31.061465 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:31.061450    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/bb1f4e62-0113-40c8-b837-8f0a75d4f2c9-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"bb1f4e62-0113-40c8-b837-8f0a75d4f2c9\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 11:21:31.061825 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:31.061809    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/bb1f4e62-0113-40c8-b837-8f0a75d4f2c9-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"bb1f4e62-0113-40c8-b837-8f0a75d4f2c9\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 11:21:31.067367 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:31.067346    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7dqkl\" (UniqueName: \"kubernetes.io/projected/bb1f4e62-0113-40c8-b837-8f0a75d4f2c9-kube-api-access-7dqkl\") pod \"prometheus-k8s-0\" (UID: \"bb1f4e62-0113-40c8-b837-8f0a75d4f2c9\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 11:21:31.098376 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:31.098321    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dced9206-c2e9-4a0d-9a21-4e5c898dcfa8" path="/var/lib/kubelet/pods/dced9206-c2e9-4a0d-9a21-4e5c898dcfa8/volumes"
May 25 11:21:31.204290 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:31.204248    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
May 25 11:21:31.339679 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:31.339639    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
May 25 11:21:31.342956 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:21:31.342930    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb1f4e62_0113_40c8_b837_8f0a75d4f2c9.slice/crio-e85ee627adf25d1519db6298650be53adc547bc47620d39f507ac1983ac5dfdd WatchSource:0}: Error finding container e85ee627adf25d1519db6298650be53adc547bc47620d39f507ac1983ac5dfdd: Status 404 returned error can't find the container with id e85ee627adf25d1519db6298650be53adc547bc47620d39f507ac1983ac5dfdd
May 25 11:21:31.838532 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:31.838496    2576 generic.go:358] "Generic (PLEG): container finished" podID="bb1f4e62-0113-40c8-b837-8f0a75d4f2c9" containerID="424d38a536e1565f5f015459f133d6f4499054719856d120dcbfff39fa553135" exitCode=0
May 25 11:21:31.838944 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:31.838546    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bb1f4e62-0113-40c8-b837-8f0a75d4f2c9","Type":"ContainerDied","Data":"424d38a536e1565f5f015459f133d6f4499054719856d120dcbfff39fa553135"}
May 25 11:21:31.838944 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:31.838571    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bb1f4e62-0113-40c8-b837-8f0a75d4f2c9","Type":"ContainerStarted","Data":"e85ee627adf25d1519db6298650be53adc547bc47620d39f507ac1983ac5dfdd"}
May 25 11:21:32.844593 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:32.844557    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bb1f4e62-0113-40c8-b837-8f0a75d4f2c9","Type":"ContainerStarted","Data":"586bec7ea0c2e49b116df568983d644310d27ce3bc07765cd9f49088b2500025"}
May 25 11:21:32.844593 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:32.844593    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bb1f4e62-0113-40c8-b837-8f0a75d4f2c9","Type":"ContainerStarted","Data":"194d8763ceb73d58773ba8c5cc0158b55cf531ae3aad7b6153693bd70f937c0d"}
May 25 11:21:32.844593 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:32.844603    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bb1f4e62-0113-40c8-b837-8f0a75d4f2c9","Type":"ContainerStarted","Data":"a3298de2e6a3c6edb538d8b2e8091cd118d6a9735f96b65bbae591e75c123916"}
May 25 11:21:32.845126 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:32.844611    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bb1f4e62-0113-40c8-b837-8f0a75d4f2c9","Type":"ContainerStarted","Data":"3d4465f96bf4f04fb589b61ace2b8eb5d99f922111e6c6aa3d36243001c29469"}
May 25 11:21:32.845126 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:32.844620    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bb1f4e62-0113-40c8-b837-8f0a75d4f2c9","Type":"ContainerStarted","Data":"61c2ccf749f85e2dabf01f3cf652c9dd9c2391e7c61bbeeafd40d171a0d35de6"}
May 25 11:21:32.845126 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:32.844628    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bb1f4e62-0113-40c8-b837-8f0a75d4f2c9","Type":"ContainerStarted","Data":"39d19cfc7bba169d49534458d4ca96856a19b377a17ee86d9fded6e6615d0cb0"}
May 25 11:21:32.880221 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:32.880177    2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.880163348 podStartE2EDuration="2.880163348s" podCreationTimestamp="2026-05-25 11:21:30 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 11:21:32.879868949 +0000 UTC m=+238.397600434" watchObservedRunningTime="2026-05-25 11:21:32.880163348 +0000 UTC m=+238.397894803"
May 25 11:21:36.205241 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:21:36.205212    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
May 25 11:22:31.205063 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:22:31.205029    2576 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
May 25 11:22:31.220392 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:22:31.220361    2576 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
May 25 11:22:32.030242 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:22:32.030217    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
May 25 11:22:35.007464 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:22:35.007428    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h6b5t_12b45353-8444-41fc-a8ed-fb691764eb63/ovn-acl-logging/0.log"
May 25 11:22:35.009741 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:22:35.009714    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h6b5t_12b45353-8444-41fc-a8ed-fb691764eb63/ovn-acl-logging/0.log"
May 25 11:22:35.017011 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:22:35.016991    2576 kubelet.go:1628] "Image garbage collection succeeded"
May 25 11:24:17.305014 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:24:17.304977    2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-68b757865b-95t57"]
May 25 11:24:17.308318 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:24:17.308298    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-68b757865b-95t57"
May 25 11:24:17.310670 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:24:17.310649    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
May 25 11:24:17.311446 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:24:17.311427    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
May 25 11:24:17.311543 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:24:17.311427    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-fhmkq\""
May 25 11:24:17.319348 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:24:17.319325    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-68b757865b-95t57"]
May 25 11:24:17.485960 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:24:17.485927    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/46277761-c4f4-4a13-b786-464b087fe73c-bound-sa-token\") pod \"cert-manager-cainjector-68b757865b-95t57\" (UID: \"46277761-c4f4-4a13-b786-464b087fe73c\") " pod="cert-manager/cert-manager-cainjector-68b757865b-95t57"
May 25 11:24:17.486190 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:24:17.485978    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvjkg\" (UniqueName: \"kubernetes.io/projected/46277761-c4f4-4a13-b786-464b087fe73c-kube-api-access-rvjkg\") pod \"cert-manager-cainjector-68b757865b-95t57\" (UID: \"46277761-c4f4-4a13-b786-464b087fe73c\") " pod="cert-manager/cert-manager-cainjector-68b757865b-95t57"
May 25 11:24:17.587526 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:24:17.587429    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-rvjkg\" (UniqueName: \"kubernetes.io/projected/46277761-c4f4-4a13-b786-464b087fe73c-kube-api-access-rvjkg\") pod \"cert-manager-cainjector-68b757865b-95t57\" (UID: \"46277761-c4f4-4a13-b786-464b087fe73c\") " pod="cert-manager/cert-manager-cainjector-68b757865b-95t57"
May 25 11:24:17.587526 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:24:17.587518    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/46277761-c4f4-4a13-b786-464b087fe73c-bound-sa-token\") pod \"cert-manager-cainjector-68b757865b-95t57\" (UID: \"46277761-c4f4-4a13-b786-464b087fe73c\") " pod="cert-manager/cert-manager-cainjector-68b757865b-95t57"
May 25 11:24:17.602364 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:24:17.602337    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/46277761-c4f4-4a13-b786-464b087fe73c-bound-sa-token\") pod \"cert-manager-cainjector-68b757865b-95t57\" (UID: \"46277761-c4f4-4a13-b786-464b087fe73c\") " pod="cert-manager/cert-manager-cainjector-68b757865b-95t57"
May 25 11:24:17.602490 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:24:17.602471    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvjkg\" (UniqueName: \"kubernetes.io/projected/46277761-c4f4-4a13-b786-464b087fe73c-kube-api-access-rvjkg\") pod \"cert-manager-cainjector-68b757865b-95t57\" (UID: \"46277761-c4f4-4a13-b786-464b087fe73c\") " pod="cert-manager/cert-manager-cainjector-68b757865b-95t57"
May 25 11:24:17.625882 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:24:17.625856    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-68b757865b-95t57"
May 25 11:24:17.759944 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:24:17.759920    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-68b757865b-95t57"]
May 25 11:24:17.761903 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:24:17.761873    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod46277761_c4f4_4a13_b786_464b087fe73c.slice/crio-f4882efcf4e2f07168b32228fcc100a3fe218a0eeda93b6a932eb9a7b2633585 WatchSource:0}: Error finding container f4882efcf4e2f07168b32228fcc100a3fe218a0eeda93b6a932eb9a7b2633585: Status 404 returned error can't find the container with id f4882efcf4e2f07168b32228fcc100a3fe218a0eeda93b6a932eb9a7b2633585
May 25 11:24:17.764013 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:24:17.763995    2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 25 11:24:18.314498 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:24:18.314464    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-68b757865b-95t57" event={"ID":"46277761-c4f4-4a13-b786-464b087fe73c","Type":"ContainerStarted","Data":"f4882efcf4e2f07168b32228fcc100a3fe218a0eeda93b6a932eb9a7b2633585"}
May 25 11:24:25.336749 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:24:25.336705    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-68b757865b-95t57" event={"ID":"46277761-c4f4-4a13-b786-464b087fe73c","Type":"ContainerStarted","Data":"4d7f6a34fd108a586f625a248a1db5e0ddcc3e8c057c931797f93c9b1d62fead"}
May 25 11:24:25.350647 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:24:25.350603    2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-68b757865b-95t57" podStartSLOduration=1.388264821 podStartE2EDuration="8.350587653s" podCreationTimestamp="2026-05-25 11:24:17 +0000 UTC" firstStartedPulling="2026-05-25 11:24:17.764127567 +0000 UTC m=+403.281859003" lastFinishedPulling="2026-05-25 11:24:24.726450397 +0000 UTC m=+410.244181835" observedRunningTime="2026-05-25 11:24:25.350551462 +0000 UTC m=+410.868282918" watchObservedRunningTime="2026-05-25 11:24:25.350587653 +0000 UTC m=+410.868319109"
May 25 11:24:34.301476 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:24:34.301441    2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-79c8d999ff-f6clj"]
May 25 11:24:34.305183 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:24:34.305158    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-79c8d999ff-f6clj"
May 25 11:24:34.307398 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:24:34.307375    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-mcmff\""
May 25 11:24:34.311577 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:24:34.311552    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-79c8d999ff-f6clj"]
May 25 11:24:34.419718 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:24:34.419659    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0990960d-0eda-4c65-ba97-1cee427cda0c-bound-sa-token\") pod \"cert-manager-79c8d999ff-f6clj\" (UID: \"0990960d-0eda-4c65-ba97-1cee427cda0c\") " pod="cert-manager/cert-manager-79c8d999ff-f6clj"
May 25 11:24:34.419895 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:24:34.419788    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6mwld\" (UniqueName: \"kubernetes.io/projected/0990960d-0eda-4c65-ba97-1cee427cda0c-kube-api-access-6mwld\") pod \"cert-manager-79c8d999ff-f6clj\" (UID: \"0990960d-0eda-4c65-ba97-1cee427cda0c\") " pod="cert-manager/cert-manager-79c8d999ff-f6clj"
May 25 11:24:34.520226 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:24:34.520189    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-6mwld\" (UniqueName: \"kubernetes.io/projected/0990960d-0eda-4c65-ba97-1cee427cda0c-kube-api-access-6mwld\") pod \"cert-manager-79c8d999ff-f6clj\" (UID: \"0990960d-0eda-4c65-ba97-1cee427cda0c\") " pod="cert-manager/cert-manager-79c8d999ff-f6clj"
May 25 11:24:34.520373 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:24:34.520256    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0990960d-0eda-4c65-ba97-1cee427cda0c-bound-sa-token\") pod \"cert-manager-79c8d999ff-f6clj\" (UID: \"0990960d-0eda-4c65-ba97-1cee427cda0c\") " pod="cert-manager/cert-manager-79c8d999ff-f6clj"
May 25 11:24:34.528570 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:24:34.528537    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0990960d-0eda-4c65-ba97-1cee427cda0c-bound-sa-token\") pod \"cert-manager-79c8d999ff-f6clj\" (UID: \"0990960d-0eda-4c65-ba97-1cee427cda0c\") " pod="cert-manager/cert-manager-79c8d999ff-f6clj"
May 25 11:24:34.528742 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:24:34.528717    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6mwld\" (UniqueName: \"kubernetes.io/projected/0990960d-0eda-4c65-ba97-1cee427cda0c-kube-api-access-6mwld\") pod \"cert-manager-79c8d999ff-f6clj\" (UID: \"0990960d-0eda-4c65-ba97-1cee427cda0c\") " pod="cert-manager/cert-manager-79c8d999ff-f6clj"
May 25 11:24:34.615331 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:24:34.615226    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-79c8d999ff-f6clj"
May 25 11:24:34.737522 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:24:34.737493    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-79c8d999ff-f6clj"]
May 25 11:24:34.740097 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:24:34.740073    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0990960d_0eda_4c65_ba97_1cee427cda0c.slice/crio-534604dde6412d20b744222db809164bab7199f90ebe39a6dd3c548333a2f3d9 WatchSource:0}: Error finding container 534604dde6412d20b744222db809164bab7199f90ebe39a6dd3c548333a2f3d9: Status 404 returned error can't find the container with id 534604dde6412d20b744222db809164bab7199f90ebe39a6dd3c548333a2f3d9
May 25 11:24:35.369050 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:24:35.369006    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-79c8d999ff-f6clj" event={"ID":"0990960d-0eda-4c65-ba97-1cee427cda0c","Type":"ContainerStarted","Data":"064c665c43c33dec71d05767333e7ecaae974af6ca951681ce5a852c8c1ce9f8"}
May 25 11:24:35.369050 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:24:35.369055    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-79c8d999ff-f6clj" event={"ID":"0990960d-0eda-4c65-ba97-1cee427cda0c","Type":"ContainerStarted","Data":"534604dde6412d20b744222db809164bab7199f90ebe39a6dd3c548333a2f3d9"}
May 25 11:24:35.387927 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:24:35.387877    2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="cert-manager/cert-manager-79c8d999ff-f6clj" podStartSLOduration=1.3878621469999999 podStartE2EDuration="1.387862147s" podCreationTimestamp="2026-05-25 11:24:34 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 11:24:35.386307178 +0000 UTC m=+420.904038629" watchObservedRunningTime="2026-05-25 11:24:35.387862147 +0000 UTC m=+420.905593603"
May 25 11:25:07.384024 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:25:07.383989    2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-757cd88c59-mqdwf"]
May 25 11:25:07.390441 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:25:07.390417    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-757cd88c59-mqdwf"
May 25 11:25:07.394106 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:25:07.394078    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-2qp7b\""
May 25 11:25:07.394340 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:25:07.394106    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
May 25 11:25:07.394340 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:25:07.394219    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
May 25 11:25:07.394340 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:25:07.394132    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
May 25 11:25:07.394340 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:25:07.394117    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
May 25 11:25:07.394340 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:25:07.394156    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
May 25 11:25:07.396354 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:25:07.396331    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-757cd88c59-mqdwf"]
May 25 11:25:07.501008 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:25:07.500969    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1315774b-398d-4228-81a3-1e9d0001ccc8-cert\") pod \"lws-controller-manager-757cd88c59-mqdwf\" (UID: \"1315774b-398d-4228-81a3-1e9d0001ccc8\") " pod="openshift-lws-operator/lws-controller-manager-757cd88c59-mqdwf"
May 25 11:25:07.501181 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:25:07.501040    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/1315774b-398d-4228-81a3-1e9d0001ccc8-metrics-cert\") pod \"lws-controller-manager-757cd88c59-mqdwf\" (UID: \"1315774b-398d-4228-81a3-1e9d0001ccc8\") " pod="openshift-lws-operator/lws-controller-manager-757cd88c59-mqdwf"
May 25 11:25:07.501181 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:25:07.501076    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzsnc\" (UniqueName: \"kubernetes.io/projected/1315774b-398d-4228-81a3-1e9d0001ccc8-kube-api-access-rzsnc\") pod \"lws-controller-manager-757cd88c59-mqdwf\" (UID: \"1315774b-398d-4228-81a3-1e9d0001ccc8\") " pod="openshift-lws-operator/lws-controller-manager-757cd88c59-mqdwf"
May 25 11:25:07.501181 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:25:07.501109    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/1315774b-398d-4228-81a3-1e9d0001ccc8-manager-config\") pod \"lws-controller-manager-757cd88c59-mqdwf\" (UID: \"1315774b-398d-4228-81a3-1e9d0001ccc8\") " pod="openshift-lws-operator/lws-controller-manager-757cd88c59-mqdwf"
May 25 11:25:07.602094 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:25:07.602052    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1315774b-398d-4228-81a3-1e9d0001ccc8-cert\") pod \"lws-controller-manager-757cd88c59-mqdwf\" (UID: \"1315774b-398d-4228-81a3-1e9d0001ccc8\") " pod="openshift-lws-operator/lws-controller-manager-757cd88c59-mqdwf"
May 25 11:25:07.602266 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:25:07.602122    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/1315774b-398d-4228-81a3-1e9d0001ccc8-metrics-cert\") pod \"lws-controller-manager-757cd88c59-mqdwf\" (UID: \"1315774b-398d-4228-81a3-1e9d0001ccc8\") " pod="openshift-lws-operator/lws-controller-manager-757cd88c59-mqdwf"
May 25 11:25:07.602266 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:25:07.602168    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-rzsnc\" (UniqueName: \"kubernetes.io/projected/1315774b-398d-4228-81a3-1e9d0001ccc8-kube-api-access-rzsnc\") pod \"lws-controller-manager-757cd88c59-mqdwf\" (UID: \"1315774b-398d-4228-81a3-1e9d0001ccc8\") " pod="openshift-lws-operator/lws-controller-manager-757cd88c59-mqdwf"
May 25 11:25:07.602266 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:25:07.602217    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/1315774b-398d-4228-81a3-1e9d0001ccc8-manager-config\") pod \"lws-controller-manager-757cd88c59-mqdwf\" (UID: \"1315774b-398d-4228-81a3-1e9d0001ccc8\") " pod="openshift-lws-operator/lws-controller-manager-757cd88c59-mqdwf"
May 25 11:25:07.602920 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:25:07.602893    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/1315774b-398d-4228-81a3-1e9d0001ccc8-manager-config\") pod \"lws-controller-manager-757cd88c59-mqdwf\" (UID: \"1315774b-398d-4228-81a3-1e9d0001ccc8\") " pod="openshift-lws-operator/lws-controller-manager-757cd88c59-mqdwf"
May 25 11:25:07.604744 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:25:07.604715    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1315774b-398d-4228-81a3-1e9d0001ccc8-cert\") pod \"lws-controller-manager-757cd88c59-mqdwf\" (UID: \"1315774b-398d-4228-81a3-1e9d0001ccc8\") " pod="openshift-lws-operator/lws-controller-manager-757cd88c59-mqdwf"
May 25 11:25:07.604846 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:25:07.604778    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/1315774b-398d-4228-81a3-1e9d0001ccc8-metrics-cert\") pod \"lws-controller-manager-757cd88c59-mqdwf\" (UID: \"1315774b-398d-4228-81a3-1e9d0001ccc8\") " pod="openshift-lws-operator/lws-controller-manager-757cd88c59-mqdwf"
May 25 11:25:07.615322 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:25:07.615299    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzsnc\" (UniqueName: \"kubernetes.io/projected/1315774b-398d-4228-81a3-1e9d0001ccc8-kube-api-access-rzsnc\") pod \"lws-controller-manager-757cd88c59-mqdwf\" (UID: \"1315774b-398d-4228-81a3-1e9d0001ccc8\") " pod="openshift-lws-operator/lws-controller-manager-757cd88c59-mqdwf"
May 25 11:25:07.701895 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:25:07.701865    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-757cd88c59-mqdwf"
May 25 11:25:07.838428 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:25:07.838399    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-757cd88c59-mqdwf"]
May 25 11:25:07.840991 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:25:07.840965    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1315774b_398d_4228_81a3_1e9d0001ccc8.slice/crio-c16e8d36caaa46f7b5e1268f66293f704982a6401bff607f325b4e31b1d45c0d WatchSource:0}: Error finding container c16e8d36caaa46f7b5e1268f66293f704982a6401bff607f325b4e31b1d45c0d: Status 404 returned error can't find the container with id c16e8d36caaa46f7b5e1268f66293f704982a6401bff607f325b4e31b1d45c0d
May 25 11:25:08.465995 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:25:08.465959    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-757cd88c59-mqdwf" event={"ID":"1315774b-398d-4228-81a3-1e9d0001ccc8","Type":"ContainerStarted","Data":"c16e8d36caaa46f7b5e1268f66293f704982a6401bff607f325b4e31b1d45c0d"}
May 25 11:25:10.472644 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:25:10.472612    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-757cd88c59-mqdwf" event={"ID":"1315774b-398d-4228-81a3-1e9d0001ccc8","Type":"ContainerStarted","Data":"0ff9c272ebc78ec6996351515067834b7f09c4e26365cc764a5f116b62a3d2a8"}
May 25 11:25:10.473000 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:25:10.472767    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-757cd88c59-mqdwf"
May 25 11:25:10.493337 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:25:10.493270    2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-757cd88c59-mqdwf" podStartSLOduration=0.937782554 podStartE2EDuration="3.493253602s" podCreationTimestamp="2026-05-25 11:25:07 +0000 UTC" firstStartedPulling="2026-05-25 11:25:07.842772585 +0000 UTC m=+453.360504018" lastFinishedPulling="2026-05-25 11:25:10.398243617 +0000 UTC m=+455.915975066" observedRunningTime="2026-05-25 11:25:10.492260671 +0000 UTC m=+456.009992127" watchObservedRunningTime="2026-05-25 11:25:10.493253602 +0000 UTC m=+456.010985059"
May 25 11:25:21.478313 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:25:21.478269    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-757cd88c59-mqdwf"
May 25 11:25:49.458898 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:25:49.458860    2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-844548ff4c-gg87l"]
May 25 11:25:49.462180 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:25:49.462164    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-844548ff4c-gg87l"
May 25 11:25:49.464564 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:25:49.464539    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
May 25 11:25:49.464564 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:25:49.464552    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
May 25 11:25:49.464801 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:25:49.464589    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
May 25 11:25:49.465226 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:25:49.465207    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-zplbd\""
May 25 11:25:49.484928 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:25:49.484902    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-844548ff4c-gg87l"]
May 25 11:25:49.540888 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:25:49.540857    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mlxdd\" (UniqueName: \"kubernetes.io/projected/fe69b762-18ab-4120-ba16-59a96c569243-kube-api-access-mlxdd\") pod \"dns-operator-controller-manager-844548ff4c-gg87l\" (UID: \"fe69b762-18ab-4120-ba16-59a96c569243\") " pod="kuadrant-system/dns-operator-controller-manager-844548ff4c-gg87l"
May 25 11:25:49.641923 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:25:49.641896    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-mlxdd\" (UniqueName: \"kubernetes.io/projected/fe69b762-18ab-4120-ba16-59a96c569243-kube-api-access-mlxdd\") pod \"dns-operator-controller-manager-844548ff4c-gg87l\" (UID: \"fe69b762-18ab-4120-ba16-59a96c569243\") " pod="kuadrant-system/dns-operator-controller-manager-844548ff4c-gg87l"
May 25 11:25:49.663913 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:25:49.663893    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mlxdd\" (UniqueName: \"kubernetes.io/projected/fe69b762-18ab-4120-ba16-59a96c569243-kube-api-access-mlxdd\") pod \"dns-operator-controller-manager-844548ff4c-gg87l\" (UID: \"fe69b762-18ab-4120-ba16-59a96c569243\") " pod="kuadrant-system/dns-operator-controller-manager-844548ff4c-gg87l"
May 25 11:25:49.771924 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:25:49.771868    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-844548ff4c-gg87l"
May 25 11:25:49.899673 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:25:49.899646    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-844548ff4c-gg87l"]
May 25 11:25:49.902343 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:25:49.902315    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe69b762_18ab_4120_ba16_59a96c569243.slice/crio-6d94b33e873bf328365e8cd6be2a9d08a8ecc745d8e1b52b71db128c9c242f66 WatchSource:0}: Error finding container 6d94b33e873bf328365e8cd6be2a9d08a8ecc745d8e1b52b71db128c9c242f66: Status 404 returned error can't find the container with id 6d94b33e873bf328365e8cd6be2a9d08a8ecc745d8e1b52b71db128c9c242f66
May 25 11:25:50.589912 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:25:50.589853    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-844548ff4c-gg87l" event={"ID":"fe69b762-18ab-4120-ba16-59a96c569243","Type":"ContainerStarted","Data":"6d94b33e873bf328365e8cd6be2a9d08a8ecc745d8e1b52b71db128c9c242f66"}
May 25 11:25:52.596826 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:25:52.596739    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-844548ff4c-gg87l" event={"ID":"fe69b762-18ab-4120-ba16-59a96c569243","Type":"ContainerStarted","Data":"0331b22caa9f8f04d6e759a4eefdf0c6ba271ff3c94118345d581f565f5646b9"}
May 25 11:25:52.597212 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:25:52.596871    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-844548ff4c-gg87l"
May 25 11:25:52.637939 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:25:52.637886    2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-844548ff4c-gg87l" podStartSLOduration=1.282681995 podStartE2EDuration="3.637871808s" podCreationTimestamp="2026-05-25 11:25:49 +0000 UTC" firstStartedPulling="2026-05-25 11:25:49.904884171 +0000 UTC m=+495.422615609" lastFinishedPulling="2026-05-25 11:25:52.260073985 +0000 UTC m=+497.777805422" observedRunningTime="2026-05-25 11:25:52.635174593 +0000 UTC m=+498.152906061" watchObservedRunningTime="2026-05-25 11:25:52.637871808 +0000 UTC m=+498.155603263"
May 25 11:25:54.978212 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:25:54.978172    2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-6c886788f8-zjgpc"]
May 25 11:25:54.981751 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:25:54.981730    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-6c886788f8-zjgpc"
May 25 11:25:54.983929 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:25:54.983904    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
May 25 11:25:54.984044 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:25:54.983916    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
May 25 11:25:54.984044 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:25:54.983916    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-rv2jq\""
May 25 11:25:54.996822 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:25:54.996802    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-6c886788f8-zjgpc"]
May 25 11:25:55.085272 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:25:55.085246    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbgr4\" (UniqueName: \"kubernetes.io/projected/0657e802-a542-4e6c-b86b-22c168006885-kube-api-access-qbgr4\") pod \"kuadrant-console-plugin-6c886788f8-zjgpc\" (UID: \"0657e802-a542-4e6c-b86b-22c168006885\") " pod="kuadrant-system/kuadrant-console-plugin-6c886788f8-zjgpc"
May 25 11:25:55.085430 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:25:55.085320    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/0657e802-a542-4e6c-b86b-22c168006885-plugin-serving-cert\") pod \"kuadrant-console-plugin-6c886788f8-zjgpc\" (UID: \"0657e802-a542-4e6c-b86b-22c168006885\") " pod="kuadrant-system/kuadrant-console-plugin-6c886788f8-zjgpc"
May 25 11:25:55.085430 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:25:55.085351    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/0657e802-a542-4e6c-b86b-22c168006885-nginx-conf\") pod \"kuadrant-console-plugin-6c886788f8-zjgpc\" (UID: \"0657e802-a542-4e6c-b86b-22c168006885\") " pod="kuadrant-system/kuadrant-console-plugin-6c886788f8-zjgpc"
May 25 11:25:55.186073 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:25:55.186045    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/0657e802-a542-4e6c-b86b-22c168006885-plugin-serving-cert\") pod \"kuadrant-console-plugin-6c886788f8-zjgpc\" (UID: \"0657e802-a542-4e6c-b86b-22c168006885\") " pod="kuadrant-system/kuadrant-console-plugin-6c886788f8-zjgpc"
May 25 11:25:55.186199 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:25:55.186089    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/0657e802-a542-4e6c-b86b-22c168006885-nginx-conf\") pod \"kuadrant-console-plugin-6c886788f8-zjgpc\" (UID: \"0657e802-a542-4e6c-b86b-22c168006885\") " pod="kuadrant-system/kuadrant-console-plugin-6c886788f8-zjgpc"
May 25 11:25:55.186199 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:25:55.186139    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-qbgr4\" (UniqueName: \"kubernetes.io/projected/0657e802-a542-4e6c-b86b-22c168006885-kube-api-access-qbgr4\") pod \"kuadrant-console-plugin-6c886788f8-zjgpc\" (UID: \"0657e802-a542-4e6c-b86b-22c168006885\") " pod="kuadrant-system/kuadrant-console-plugin-6c886788f8-zjgpc"
May 25 11:25:55.186762 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:25:55.186738    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/0657e802-a542-4e6c-b86b-22c168006885-nginx-conf\") pod \"kuadrant-console-plugin-6c886788f8-zjgpc\" (UID: \"0657e802-a542-4e6c-b86b-22c168006885\") " pod="kuadrant-system/kuadrant-console-plugin-6c886788f8-zjgpc"
May 25 11:25:55.188485 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:25:55.188467    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/0657e802-a542-4e6c-b86b-22c168006885-plugin-serving-cert\") pod \"kuadrant-console-plugin-6c886788f8-zjgpc\" (UID: \"0657e802-a542-4e6c-b86b-22c168006885\") " pod="kuadrant-system/kuadrant-console-plugin-6c886788f8-zjgpc"
May 25 11:25:55.194547 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:25:55.194524    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbgr4\" (UniqueName: \"kubernetes.io/projected/0657e802-a542-4e6c-b86b-22c168006885-kube-api-access-qbgr4\") pod \"kuadrant-console-plugin-6c886788f8-zjgpc\" (UID: \"0657e802-a542-4e6c-b86b-22c168006885\") " pod="kuadrant-system/kuadrant-console-plugin-6c886788f8-zjgpc"
May 25 11:25:55.291137 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:25:55.291085    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-6c886788f8-zjgpc"
May 25 11:25:55.412011 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:25:55.411985    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-6c886788f8-zjgpc"]
May 25 11:25:55.414226 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:25:55.414194    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0657e802_a542_4e6c_b86b_22c168006885.slice/crio-2e07a53b8772a9b893a7175b0b4d8bf84097215c5934fdf5c7dcca0ff5a3175e WatchSource:0}: Error finding container 2e07a53b8772a9b893a7175b0b4d8bf84097215c5934fdf5c7dcca0ff5a3175e: Status 404 returned error can't find the container with id 2e07a53b8772a9b893a7175b0b4d8bf84097215c5934fdf5c7dcca0ff5a3175e
May 25 11:25:55.608203 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:25:55.608134    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-6c886788f8-zjgpc" event={"ID":"0657e802-a542-4e6c-b86b-22c168006885","Type":"ContainerStarted","Data":"2e07a53b8772a9b893a7175b0b4d8bf84097215c5934fdf5c7dcca0ff5a3175e"}
May 25 11:26:03.602411 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:26:03.602336    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-844548ff4c-gg87l"
May 25 11:26:03.643859 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:26:03.643824    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-6c886788f8-zjgpc" event={"ID":"0657e802-a542-4e6c-b86b-22c168006885","Type":"ContainerStarted","Data":"c076bd047c50774c88d9a00fb16ebe9c11cbf35c4219f1aafa8e558aeca5abf0"}
May 25 11:26:03.663568 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:26:03.663517    2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-6c886788f8-zjgpc" podStartSLOduration=1.856103724 podStartE2EDuration="9.663502566s" podCreationTimestamp="2026-05-25 11:25:54 +0000 UTC" firstStartedPulling="2026-05-25 11:25:55.415607721 +0000 UTC m=+500.933339154" lastFinishedPulling="2026-05-25 11:26:03.223006563 +0000 UTC m=+508.740737996" observedRunningTime="2026-05-25 11:26:03.661310829 +0000 UTC m=+509.179042285" watchObservedRunningTime="2026-05-25 11:26:03.663502566 +0000 UTC m=+509.181234067"
May 25 11:26:38.408165 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:26:38.408075    2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-64c8f475fb-hfw47"]
May 25 11:26:38.434873 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:26:38.434841    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c8f475fb-hfw47"]
May 25 11:26:38.435030 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:26:38.434956    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c8f475fb-hfw47"
May 25 11:26:38.437339 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:26:38.437316    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
May 25 11:26:38.502477 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:26:38.502443    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c8f475fb-hfw47"]
May 25 11:26:38.536424 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:26:38.536394    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/9a532e4a-5d16-452d-814f-56a4ef304778-config-file\") pod \"limitador-limitador-64c8f475fb-hfw47\" (UID: \"9a532e4a-5d16-452d-814f-56a4ef304778\") " pod="kuadrant-system/limitador-limitador-64c8f475fb-hfw47"
May 25 11:26:38.536598 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:26:38.536486    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v657p\" (UniqueName: \"kubernetes.io/projected/9a532e4a-5d16-452d-814f-56a4ef304778-kube-api-access-v657p\") pod \"limitador-limitador-64c8f475fb-hfw47\" (UID: \"9a532e4a-5d16-452d-814f-56a4ef304778\") " pod="kuadrant-system/limitador-limitador-64c8f475fb-hfw47"
May 25 11:26:38.637897 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:26:38.637863    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/9a532e4a-5d16-452d-814f-56a4ef304778-config-file\") pod \"limitador-limitador-64c8f475fb-hfw47\" (UID: \"9a532e4a-5d16-452d-814f-56a4ef304778\") " pod="kuadrant-system/limitador-limitador-64c8f475fb-hfw47"
May 25 11:26:38.638072 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:26:38.637914    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-v657p\" (UniqueName: \"kubernetes.io/projected/9a532e4a-5d16-452d-814f-56a4ef304778-kube-api-access-v657p\") pod \"limitador-limitador-64c8f475fb-hfw47\" (UID: \"9a532e4a-5d16-452d-814f-56a4ef304778\") " pod="kuadrant-system/limitador-limitador-64c8f475fb-hfw47"
May 25 11:26:38.638511 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:26:38.638494    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/9a532e4a-5d16-452d-814f-56a4ef304778-config-file\") pod \"limitador-limitador-64c8f475fb-hfw47\" (UID: \"9a532e4a-5d16-452d-814f-56a4ef304778\") " pod="kuadrant-system/limitador-limitador-64c8f475fb-hfw47"
May 25 11:26:38.645728 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:26:38.645702    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v657p\" (UniqueName: \"kubernetes.io/projected/9a532e4a-5d16-452d-814f-56a4ef304778-kube-api-access-v657p\") pod \"limitador-limitador-64c8f475fb-hfw47\" (UID: \"9a532e4a-5d16-452d-814f-56a4ef304778\") " pod="kuadrant-system/limitador-limitador-64c8f475fb-hfw47"
May 25 11:26:38.745491 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:26:38.745453    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c8f475fb-hfw47"
May 25 11:26:38.886410 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:26:38.886383    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c8f475fb-hfw47"]
May 25 11:26:38.888677 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:26:38.888648    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a532e4a_5d16_452d_814f_56a4ef304778.slice/crio-44a099d5f07f717fb8212f2645632e0b2bbefea791830b506c1a6245abc0fc12 WatchSource:0}: Error finding container 44a099d5f07f717fb8212f2645632e0b2bbefea791830b506c1a6245abc0fc12: Status 404 returned error can't find the container with id 44a099d5f07f717fb8212f2645632e0b2bbefea791830b506c1a6245abc0fc12
May 25 11:26:39.428648 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:26:39.428612    2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-79cbc94b89-ccf9k"]
May 25 11:26:39.433394 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:26:39.433374    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-79cbc94b89-ccf9k"
May 25 11:26:39.435458 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:26:39.435435    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-sw2nc\""
May 25 11:26:39.439224 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:26:39.439195    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-79cbc94b89-ccf9k"]
May 25 11:26:39.445616 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:26:39.445591    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hnckm\" (UniqueName: \"kubernetes.io/projected/e89617e4-9eda-4e71-84a6-40edfa90e831-kube-api-access-hnckm\") pod \"authorino-79cbc94b89-ccf9k\" (UID: \"e89617e4-9eda-4e71-84a6-40edfa90e831\") " pod="kuadrant-system/authorino-79cbc94b89-ccf9k"
May 25 11:26:39.546450 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:26:39.546417    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-hnckm\" (UniqueName: \"kubernetes.io/projected/e89617e4-9eda-4e71-84a6-40edfa90e831-kube-api-access-hnckm\") pod \"authorino-79cbc94b89-ccf9k\" (UID: \"e89617e4-9eda-4e71-84a6-40edfa90e831\") " pod="kuadrant-system/authorino-79cbc94b89-ccf9k"
May 25 11:26:39.554291 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:26:39.554260    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hnckm\" (UniqueName: \"kubernetes.io/projected/e89617e4-9eda-4e71-84a6-40edfa90e831-kube-api-access-hnckm\") pod \"authorino-79cbc94b89-ccf9k\" (UID: \"e89617e4-9eda-4e71-84a6-40edfa90e831\") " pod="kuadrant-system/authorino-79cbc94b89-ccf9k"
May 25 11:26:39.743868 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:26:39.743783    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-79cbc94b89-ccf9k"
May 25 11:26:39.758516 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:26:39.758483    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c8f475fb-hfw47" event={"ID":"9a532e4a-5d16-452d-814f-56a4ef304778","Type":"ContainerStarted","Data":"44a099d5f07f717fb8212f2645632e0b2bbefea791830b506c1a6245abc0fc12"}
May 25 11:26:39.871641 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:26:39.871606    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-79cbc94b89-ccf9k"]
May 25 11:26:39.875083 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:26:39.875060    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode89617e4_9eda_4e71_84a6_40edfa90e831.slice/crio-5eacb5d95a9aa404b31bcba48b1d78c32ff5d71fa758e942563ea970a30c5aa4 WatchSource:0}: Error finding container 5eacb5d95a9aa404b31bcba48b1d78c32ff5d71fa758e942563ea970a30c5aa4: Status 404 returned error can't find the container with id 5eacb5d95a9aa404b31bcba48b1d78c32ff5d71fa758e942563ea970a30c5aa4
May 25 11:26:40.763885 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:26:40.763844    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-79cbc94b89-ccf9k" event={"ID":"e89617e4-9eda-4e71-84a6-40edfa90e831","Type":"ContainerStarted","Data":"5eacb5d95a9aa404b31bcba48b1d78c32ff5d71fa758e942563ea970a30c5aa4"}
May 25 11:26:41.769538 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:26:41.769497    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c8f475fb-hfw47" event={"ID":"9a532e4a-5d16-452d-814f-56a4ef304778","Type":"ContainerStarted","Data":"e471a0d837a1640a77d9f44f5890319e1a2acbf092cb43e4a1c153419a241ccc"}
May 25 11:26:41.770019 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:26:41.769631    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-64c8f475fb-hfw47"
May 25 11:26:41.790408 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:26:41.790174    2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-64c8f475fb-hfw47" podStartSLOduration=1.062676666 podStartE2EDuration="3.790145758s" podCreationTimestamp="2026-05-25 11:26:38 +0000 UTC" firstStartedPulling="2026-05-25 11:26:38.890759053 +0000 UTC m=+544.408490490" lastFinishedPulling="2026-05-25 11:26:41.618228144 +0000 UTC m=+547.135959582" observedRunningTime="2026-05-25 11:26:41.789375223 +0000 UTC m=+547.307106680" watchObservedRunningTime="2026-05-25 11:26:41.790145758 +0000 UTC m=+547.307877215"
May 25 11:26:43.779794 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:26:43.779750    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-79cbc94b89-ccf9k" event={"ID":"e89617e4-9eda-4e71-84a6-40edfa90e831","Type":"ContainerStarted","Data":"fd0f113d847576558a9449c9cf992e7f2e4d30becae4873eb35c932b528df545"}
May 25 11:26:43.797698 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:26:43.797644    2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kuadrant-system/authorino-79cbc94b89-ccf9k" podStartSLOduration=1.468185494 podStartE2EDuration="4.797629158s" podCreationTimestamp="2026-05-25 11:26:39 +0000 UTC" firstStartedPulling="2026-05-25 11:26:39.876796652 +0000 UTC m=+545.394528091" lastFinishedPulling="2026-05-25 11:26:43.206240308 +0000 UTC m=+548.723971755" observedRunningTime="2026-05-25 11:26:43.796136498 +0000 UTC m=+549.313868014" watchObservedRunningTime="2026-05-25 11:26:43.797629158 +0000 UTC m=+549.315360613"
May 25 11:26:52.775028 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:26:52.774997    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-64c8f475fb-hfw47"
May 25 11:26:54.668429 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:26:54.668393    2576 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c8f475fb-hfw47"]
May 25 11:26:54.668847 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:26:54.668618    2576 kuberuntime_container.go:864] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-64c8f475fb-hfw47" podUID="9a532e4a-5d16-452d-814f-56a4ef304778" containerName="limitador" containerID="cri-o://e471a0d837a1640a77d9f44f5890319e1a2acbf092cb43e4a1c153419a241ccc" gracePeriod=30
May 25 11:26:55.216667 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:26:55.216644    2576 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c8f475fb-hfw47"
May 25 11:26:55.291007 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:26:55.290928    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/9a532e4a-5d16-452d-814f-56a4ef304778-config-file\") pod \"9a532e4a-5d16-452d-814f-56a4ef304778\" (UID: \"9a532e4a-5d16-452d-814f-56a4ef304778\") "
May 25 11:26:55.291151 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:26:55.291027    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v657p\" (UniqueName: \"kubernetes.io/projected/9a532e4a-5d16-452d-814f-56a4ef304778-kube-api-access-v657p\") pod \"9a532e4a-5d16-452d-814f-56a4ef304778\" (UID: \"9a532e4a-5d16-452d-814f-56a4ef304778\") "
May 25 11:26:55.291360 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:26:55.291334    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a532e4a-5d16-452d-814f-56a4ef304778-config-file" (OuterVolumeSpecName: "config-file") pod "9a532e4a-5d16-452d-814f-56a4ef304778" (UID: "9a532e4a-5d16-452d-814f-56a4ef304778"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 11:26:55.293123 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:26:55.293105    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a532e4a-5d16-452d-814f-56a4ef304778-kube-api-access-v657p" (OuterVolumeSpecName: "kube-api-access-v657p") pod "9a532e4a-5d16-452d-814f-56a4ef304778" (UID: "9a532e4a-5d16-452d-814f-56a4ef304778"). InnerVolumeSpecName "kube-api-access-v657p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 11:26:55.391716 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:26:55.391686    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v657p\" (UniqueName: \"kubernetes.io/projected/9a532e4a-5d16-452d-814f-56a4ef304778-kube-api-access-v657p\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
May 25 11:26:55.391716 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:26:55.391715    2576 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/9a532e4a-5d16-452d-814f-56a4ef304778-config-file\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
May 25 11:26:55.821873 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:26:55.821837    2576 generic.go:358] "Generic (PLEG): container finished" podID="9a532e4a-5d16-452d-814f-56a4ef304778" containerID="e471a0d837a1640a77d9f44f5890319e1a2acbf092cb43e4a1c153419a241ccc" exitCode=0
May 25 11:26:55.822387 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:26:55.821904    2576 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c8f475fb-hfw47"
May 25 11:26:55.822387 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:26:55.821924    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c8f475fb-hfw47" event={"ID":"9a532e4a-5d16-452d-814f-56a4ef304778","Type":"ContainerDied","Data":"e471a0d837a1640a77d9f44f5890319e1a2acbf092cb43e4a1c153419a241ccc"}
May 25 11:26:55.822387 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:26:55.821962    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c8f475fb-hfw47" event={"ID":"9a532e4a-5d16-452d-814f-56a4ef304778","Type":"ContainerDied","Data":"44a099d5f07f717fb8212f2645632e0b2bbefea791830b506c1a6245abc0fc12"}
May 25 11:26:55.822387 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:26:55.821978    2576 scope.go:117] "RemoveContainer" containerID="e471a0d837a1640a77d9f44f5890319e1a2acbf092cb43e4a1c153419a241ccc"
May 25 11:26:55.830324 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:26:55.830300    2576 scope.go:117] "RemoveContainer" containerID="e471a0d837a1640a77d9f44f5890319e1a2acbf092cb43e4a1c153419a241ccc"
May 25 11:26:55.830569 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:26:55.830551    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e471a0d837a1640a77d9f44f5890319e1a2acbf092cb43e4a1c153419a241ccc\": container with ID starting with e471a0d837a1640a77d9f44f5890319e1a2acbf092cb43e4a1c153419a241ccc not found: ID does not exist" containerID="e471a0d837a1640a77d9f44f5890319e1a2acbf092cb43e4a1c153419a241ccc"
May 25 11:26:55.830636 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:26:55.830583    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e471a0d837a1640a77d9f44f5890319e1a2acbf092cb43e4a1c153419a241ccc"} err="failed to get container status \"e471a0d837a1640a77d9f44f5890319e1a2acbf092cb43e4a1c153419a241ccc\": rpc error: code = NotFound desc = could not find container \"e471a0d837a1640a77d9f44f5890319e1a2acbf092cb43e4a1c153419a241ccc\": container with ID starting with e471a0d837a1640a77d9f44f5890319e1a2acbf092cb43e4a1c153419a241ccc not found: ID does not exist"
May 25 11:26:55.842803 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:26:55.842782    2576 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c8f475fb-hfw47"]
May 25 11:26:55.846450 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:26:55.846431    2576 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-64c8f475fb-hfw47"]
May 25 11:26:57.098662 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:26:57.098629    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9a532e4a-5d16-452d-814f-56a4ef304778" path="/var/lib/kubelet/pods/9a532e4a-5d16-452d-814f-56a4ef304778/volumes"
May 25 11:27:05.723644 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:27:05.723611    2576 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-79cbc94b89-ccf9k"]
May 25 11:27:05.724002 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:27:05.723869    2576 kuberuntime_container.go:864] "Killing container with a grace period" pod="kuadrant-system/authorino-79cbc94b89-ccf9k" podUID="e89617e4-9eda-4e71-84a6-40edfa90e831" containerName="authorino" containerID="cri-o://fd0f113d847576558a9449c9cf992e7f2e4d30becae4873eb35c932b528df545" gracePeriod=30
May 25 11:27:05.859493 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:27:05.859463    2576 generic.go:358] "Generic (PLEG): container finished" podID="e89617e4-9eda-4e71-84a6-40edfa90e831" containerID="fd0f113d847576558a9449c9cf992e7f2e4d30becae4873eb35c932b528df545" exitCode=0
May 25 11:27:05.859595 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:27:05.859506    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-79cbc94b89-ccf9k" event={"ID":"e89617e4-9eda-4e71-84a6-40edfa90e831","Type":"ContainerDied","Data":"fd0f113d847576558a9449c9cf992e7f2e4d30becae4873eb35c932b528df545"}
May 25 11:27:05.968750 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:27:05.968729    2576 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-79cbc94b89-ccf9k"
May 25 11:27:06.081897 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:27:06.081835    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hnckm\" (UniqueName: \"kubernetes.io/projected/e89617e4-9eda-4e71-84a6-40edfa90e831-kube-api-access-hnckm\") pod \"e89617e4-9eda-4e71-84a6-40edfa90e831\" (UID: \"e89617e4-9eda-4e71-84a6-40edfa90e831\") "
May 25 11:27:06.083819 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:27:06.083788    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e89617e4-9eda-4e71-84a6-40edfa90e831-kube-api-access-hnckm" (OuterVolumeSpecName: "kube-api-access-hnckm") pod "e89617e4-9eda-4e71-84a6-40edfa90e831" (UID: "e89617e4-9eda-4e71-84a6-40edfa90e831"). InnerVolumeSpecName "kube-api-access-hnckm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 11:27:06.183330 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:27:06.183306    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hnckm\" (UniqueName: \"kubernetes.io/projected/e89617e4-9eda-4e71-84a6-40edfa90e831-kube-api-access-hnckm\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
May 25 11:27:06.864149 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:27:06.864118    2576 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-79cbc94b89-ccf9k"
May 25 11:27:06.864149 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:27:06.864140    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-79cbc94b89-ccf9k" event={"ID":"e89617e4-9eda-4e71-84a6-40edfa90e831","Type":"ContainerDied","Data":"5eacb5d95a9aa404b31bcba48b1d78c32ff5d71fa758e942563ea970a30c5aa4"}
May 25 11:27:06.864629 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:27:06.864180    2576 scope.go:117] "RemoveContainer" containerID="fd0f113d847576558a9449c9cf992e7f2e4d30becae4873eb35c932b528df545"
May 25 11:27:06.884350 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:27:06.884328    2576 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-79cbc94b89-ccf9k"]
May 25 11:27:06.888667 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:27:06.888646    2576 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-79cbc94b89-ccf9k"]
May 25 11:27:07.097601 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:27:07.097569    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e89617e4-9eda-4e71-84a6-40edfa90e831" path="/var/lib/kubelet/pods/e89617e4-9eda-4e71-84a6-40edfa90e831/volumes"
May 25 11:27:14.391876 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:27:14.391839    2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-798958bb55-9h57j"]
May 25 11:27:14.392417 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:27:14.392353    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e89617e4-9eda-4e71-84a6-40edfa90e831" containerName="authorino"
May 25 11:27:14.392417 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:27:14.392374    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="e89617e4-9eda-4e71-84a6-40edfa90e831" containerName="authorino"
May 25 11:27:14.392417 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:27:14.392405    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a532e4a-5d16-452d-814f-56a4ef304778" containerName="limitador"
May 25 11:27:14.392417 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:27:14.392413    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a532e4a-5d16-452d-814f-56a4ef304778" containerName="limitador"
May 25 11:27:14.392631 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:27:14.392484    2576 memory_manager.go:356] "RemoveStaleState removing state" podUID="9a532e4a-5d16-452d-814f-56a4ef304778" containerName="limitador"
May 25 11:27:14.392631 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:27:14.392499    2576 memory_manager.go:356] "RemoveStaleState removing state" podUID="e89617e4-9eda-4e71-84a6-40edfa90e831" containerName="authorino"
May 25 11:27:14.396883 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:27:14.396861    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-798958bb55-9h57j"
May 25 11:27:14.406607 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:27:14.406574    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
May 25 11:27:14.406733 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:27:14.406670    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-m7bds\""
May 25 11:27:14.406830 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:27:14.406811    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
May 25 11:27:14.408504 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:27:14.408485    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
May 25 11:27:14.408684 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:27:14.408659    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
May 25 11:27:14.420569 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:27:14.420545    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-798958bb55-9h57j"]
May 25 11:27:14.550526 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:27:14.550490    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8j5ht\" (UniqueName: \"kubernetes.io/projected/bb525512-86d4-401a-a262-add91f2948a7-kube-api-access-8j5ht\") pod \"istiod-openshift-gateway-798958bb55-9h57j\" (UID: \"bb525512-86d4-401a-a262-add91f2948a7\") " pod="openshift-ingress/istiod-openshift-gateway-798958bb55-9h57j"
May 25 11:27:14.550689 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:27:14.550544    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/bb525512-86d4-401a-a262-add91f2948a7-istio-kubeconfig\") pod \"istiod-openshift-gateway-798958bb55-9h57j\" (UID: \"bb525512-86d4-401a-a262-add91f2948a7\") " pod="openshift-ingress/istiod-openshift-gateway-798958bb55-9h57j"
May 25 11:27:14.550689 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:27:14.550613    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb525512-86d4-401a-a262-add91f2948a7-istio-token\") pod \"istiod-openshift-gateway-798958bb55-9h57j\" (UID: \"bb525512-86d4-401a-a262-add91f2948a7\") " pod="openshift-ingress/istiod-openshift-gateway-798958bb55-9h57j"
May 25 11:27:14.550689 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:27:14.550632    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/bb525512-86d4-401a-a262-add91f2948a7-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-798958bb55-9h57j\" (UID: \"bb525512-86d4-401a-a262-add91f2948a7\") " pod="openshift-ingress/istiod-openshift-gateway-798958bb55-9h57j"
May 25 11:27:14.550689 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:27:14.550661    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/bb525512-86d4-401a-a262-add91f2948a7-cacerts\") pod \"istiod-openshift-gateway-798958bb55-9h57j\" (UID: \"bb525512-86d4-401a-a262-add91f2948a7\") " pod="openshift-ingress/istiod-openshift-gateway-798958bb55-9h57j"
May 25 11:27:14.550890 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:27:14.550721    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb525512-86d4-401a-a262-add91f2948a7-local-certs\") pod \"istiod-openshift-gateway-798958bb55-9h57j\" (UID: \"bb525512-86d4-401a-a262-add91f2948a7\") " pod="openshift-ingress/istiod-openshift-gateway-798958bb55-9h57j"
May 25 11:27:14.550890 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:27:14.550783    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/bb525512-86d4-401a-a262-add91f2948a7-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-798958bb55-9h57j\" (UID: \"bb525512-86d4-401a-a262-add91f2948a7\") " pod="openshift-ingress/istiod-openshift-gateway-798958bb55-9h57j"
May 25 11:27:14.651657 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:27:14.651598    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/bb525512-86d4-401a-a262-add91f2948a7-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-798958bb55-9h57j\" (UID: \"bb525512-86d4-401a-a262-add91f2948a7\") " pod="openshift-ingress/istiod-openshift-gateway-798958bb55-9h57j"
May 25 11:27:14.651760 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:27:14.651656    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-8j5ht\" (UniqueName: \"kubernetes.io/projected/bb525512-86d4-401a-a262-add91f2948a7-kube-api-access-8j5ht\") pod \"istiod-openshift-gateway-798958bb55-9h57j\" (UID: \"bb525512-86d4-401a-a262-add91f2948a7\") " pod="openshift-ingress/istiod-openshift-gateway-798958bb55-9h57j"
May 25 11:27:14.651760 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:27:14.651679    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/bb525512-86d4-401a-a262-add91f2948a7-istio-kubeconfig\") pod \"istiod-openshift-gateway-798958bb55-9h57j\" (UID: \"bb525512-86d4-401a-a262-add91f2948a7\") " pod="openshift-ingress/istiod-openshift-gateway-798958bb55-9h57j"
May 25 11:27:14.651760 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:27:14.651702    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb525512-86d4-401a-a262-add91f2948a7-istio-token\") pod \"istiod-openshift-gateway-798958bb55-9h57j\" (UID: \"bb525512-86d4-401a-a262-add91f2948a7\") " pod="openshift-ingress/istiod-openshift-gateway-798958bb55-9h57j"
May 25 11:27:14.651760 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:27:14.651718    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/bb525512-86d4-401a-a262-add91f2948a7-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-798958bb55-9h57j\" (UID: \"bb525512-86d4-401a-a262-add91f2948a7\") " pod="openshift-ingress/istiod-openshift-gateway-798958bb55-9h57j"
May 25 11:27:14.651760 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:27:14.651739    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/bb525512-86d4-401a-a262-add91f2948a7-cacerts\") pod \"istiod-openshift-gateway-798958bb55-9h57j\" (UID: \"bb525512-86d4-401a-a262-add91f2948a7\") " pod="openshift-ingress/istiod-openshift-gateway-798958bb55-9h57j"
May 25 11:27:14.651998 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:27:14.651762    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb525512-86d4-401a-a262-add91f2948a7-local-certs\") pod \"istiod-openshift-gateway-798958bb55-9h57j\" (UID: \"bb525512-86d4-401a-a262-add91f2948a7\") " pod="openshift-ingress/istiod-openshift-gateway-798958bb55-9h57j"
May 25 11:27:14.652594 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:27:14.652566    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/bb525512-86d4-401a-a262-add91f2948a7-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-798958bb55-9h57j\" (UID: \"bb525512-86d4-401a-a262-add91f2948a7\") " pod="openshift-ingress/istiod-openshift-gateway-798958bb55-9h57j"
May 25 11:27:14.654060 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:27:14.654038    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/bb525512-86d4-401a-a262-add91f2948a7-istio-kubeconfig\") pod \"istiod-openshift-gateway-798958bb55-9h57j\" (UID: \"bb525512-86d4-401a-a262-add91f2948a7\") " pod="openshift-ingress/istiod-openshift-gateway-798958bb55-9h57j"
May 25 11:27:14.654161 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:27:14.654139    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/bb525512-86d4-401a-a262-add91f2948a7-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-798958bb55-9h57j\" (UID: \"bb525512-86d4-401a-a262-add91f2948a7\") " pod="openshift-ingress/istiod-openshift-gateway-798958bb55-9h57j"
May 25 11:27:14.654161 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:27:14.654147    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb525512-86d4-401a-a262-add91f2948a7-local-certs\") pod \"istiod-openshift-gateway-798958bb55-9h57j\" (UID: \"bb525512-86d4-401a-a262-add91f2948a7\") " pod="openshift-ingress/istiod-openshift-gateway-798958bb55-9h57j"
May 25 11:27:14.654231 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:27:14.654172    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/bb525512-86d4-401a-a262-add91f2948a7-cacerts\") pod \"istiod-openshift-gateway-798958bb55-9h57j\" (UID: \"bb525512-86d4-401a-a262-add91f2948a7\") " pod="openshift-ingress/istiod-openshift-gateway-798958bb55-9h57j"
May 25 11:27:14.672251 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:27:14.672228    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb525512-86d4-401a-a262-add91f2948a7-istio-token\") pod \"istiod-openshift-gateway-798958bb55-9h57j\" (UID: \"bb525512-86d4-401a-a262-add91f2948a7\") " pod="openshift-ingress/istiod-openshift-gateway-798958bb55-9h57j"
May 25 11:27:14.672520 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:27:14.672504    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8j5ht\" (UniqueName: \"kubernetes.io/projected/bb525512-86d4-401a-a262-add91f2948a7-kube-api-access-8j5ht\") pod \"istiod-openshift-gateway-798958bb55-9h57j\" (UID: \"bb525512-86d4-401a-a262-add91f2948a7\") " pod="openshift-ingress/istiod-openshift-gateway-798958bb55-9h57j"
May 25 11:27:14.710879 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:27:14.710861    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-798958bb55-9h57j"
May 25 11:27:14.843127 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:27:14.843093    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-798958bb55-9h57j"]
May 25 11:27:14.846209 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:27:14.846185    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb525512_86d4_401a_a262_add91f2948a7.slice/crio-97ba02de05d9480c34b755e83abcb95ef0fe184308b1379feee23ae770caf2fb WatchSource:0}: Error finding container 97ba02de05d9480c34b755e83abcb95ef0fe184308b1379feee23ae770caf2fb: Status 404 returned error can't find the container with id 97ba02de05d9480c34b755e83abcb95ef0fe184308b1379feee23ae770caf2fb
May 25 11:27:14.896447 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:27:14.896398    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-798958bb55-9h57j" event={"ID":"bb525512-86d4-401a-a262-add91f2948a7","Type":"ContainerStarted","Data":"97ba02de05d9480c34b755e83abcb95ef0fe184308b1379feee23ae770caf2fb"}
May 25 11:27:17.525940 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:27:17.525900    2576 kubelet_resources.go:45] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
May 25 11:27:17.526258 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:27:17.525981    2576 kubelet_resources.go:45] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
May 25 11:27:17.910781 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:27:17.910685    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-798958bb55-9h57j" event={"ID":"bb525512-86d4-401a-a262-add91f2948a7","Type":"ContainerStarted","Data":"df51c0bcdf634c3c834d720a6a19d7ef969c3eeff8837b79a22162acbc254dfc"}
May 25 11:27:17.910781 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:27:17.910756    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-798958bb55-9h57j"
May 25 11:27:17.954483 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:27:17.954432    2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-798958bb55-9h57j" podStartSLOduration=1.276863095 podStartE2EDuration="3.95441423s" podCreationTimestamp="2026-05-25 11:27:14 +0000 UTC" firstStartedPulling="2026-05-25 11:27:14.848074086 +0000 UTC m=+580.365805521" lastFinishedPulling="2026-05-25 11:27:17.525625222 +0000 UTC m=+583.043356656" observedRunningTime="2026-05-25 11:27:17.952606237 +0000 UTC m=+583.470337695" watchObservedRunningTime="2026-05-25 11:27:17.95441423 +0000 UTC m=+583.472145686"
May 25 11:27:18.917495 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:27:18.917448    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-798958bb55-9h57j"
May 25 11:27:35.035420 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:27:35.035394    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h6b5t_12b45353-8444-41fc-a8ed-fb691764eb63/ovn-acl-logging/0.log"
May 25 11:27:35.035835 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:27:35.035482    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h6b5t_12b45353-8444-41fc-a8ed-fb691764eb63/ovn-acl-logging/0.log"
May 25 11:29:25.885569 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:29:25.885488    2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-57fcb7c57d8j8t8"]
May 25 11:29:25.888993 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:29:25.888972    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-57fcb7c57d8j8t8"
May 25 11:29:25.892045 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:29:25.892023    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
May 25 11:29:25.892144 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:29:25.892050    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
May 25 11:29:25.892144 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:29:25.892050    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"gw-sec0c69dceeb48768325d1a53a749e65786-kserve-self-signed-certs\""
May 25 11:29:25.892251 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:29:25.892033    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-5cxfn\""
May 25 11:29:25.900054 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:29:25.900031    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-57fcb7c57d8j8t8"]
May 25 11:29:25.915176 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:29:25.915152    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rc9n\" (UniqueName: \"kubernetes.io/projected/1dfc4e23-6d34-4fe2-af30-dc81eaaf9811-kube-api-access-2rc9n\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-57fcb7c57d8j8t8\" (UID: \"1dfc4e23-6d34-4fe2-af30-dc81eaaf9811\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-57fcb7c57d8j8t8"
May 25 11:29:25.915300 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:29:25.915196    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1dfc4e23-6d34-4fe2-af30-dc81eaaf9811-model-cache\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-57fcb7c57d8j8t8\" (UID: \"1dfc4e23-6d34-4fe2-af30-dc81eaaf9811\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-57fcb7c57d8j8t8"
May 25 11:29:25.915379 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:29:25.915302    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1dfc4e23-6d34-4fe2-af30-dc81eaaf9811-tls-certs\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-57fcb7c57d8j8t8\" (UID: \"1dfc4e23-6d34-4fe2-af30-dc81eaaf9811\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-57fcb7c57d8j8t8"
May 25 11:29:25.915379 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:29:25.915336    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1dfc4e23-6d34-4fe2-af30-dc81eaaf9811-kserve-provision-location\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-57fcb7c57d8j8t8\" (UID: \"1dfc4e23-6d34-4fe2-af30-dc81eaaf9811\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-57fcb7c57d8j8t8"
May 25 11:29:25.915379 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:29:25.915375    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1dfc4e23-6d34-4fe2-af30-dc81eaaf9811-dshm\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-57fcb7c57d8j8t8\" (UID: \"1dfc4e23-6d34-4fe2-af30-dc81eaaf9811\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-57fcb7c57d8j8t8"
May 25 11:29:25.915532 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:29:25.915431    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1dfc4e23-6d34-4fe2-af30-dc81eaaf9811-tmp-dir\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-57fcb7c57d8j8t8\" (UID: \"1dfc4e23-6d34-4fe2-af30-dc81eaaf9811\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-57fcb7c57d8j8t8"
May 25 11:29:25.915532 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:29:25.915458    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1dfc4e23-6d34-4fe2-af30-dc81eaaf9811-home\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-57fcb7c57d8j8t8\" (UID: \"1dfc4e23-6d34-4fe2-af30-dc81eaaf9811\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-57fcb7c57d8j8t8"
May 25 11:29:26.016658 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:29:26.016624    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1dfc4e23-6d34-4fe2-af30-dc81eaaf9811-tls-certs\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-57fcb7c57d8j8t8\" (UID: \"1dfc4e23-6d34-4fe2-af30-dc81eaaf9811\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-57fcb7c57d8j8t8"
May 25 11:29:26.016835 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:29:26.016666    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1dfc4e23-6d34-4fe2-af30-dc81eaaf9811-kserve-provision-location\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-57fcb7c57d8j8t8\" (UID: \"1dfc4e23-6d34-4fe2-af30-dc81eaaf9811\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-57fcb7c57d8j8t8"
May 25 11:29:26.016835 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:29:26.016693    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1dfc4e23-6d34-4fe2-af30-dc81eaaf9811-dshm\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-57fcb7c57d8j8t8\" (UID: \"1dfc4e23-6d34-4fe2-af30-dc81eaaf9811\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-57fcb7c57d8j8t8"
May 25 11:29:26.016835 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:29:26.016740    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1dfc4e23-6d34-4fe2-af30-dc81eaaf9811-tmp-dir\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-57fcb7c57d8j8t8\" (UID: \"1dfc4e23-6d34-4fe2-af30-dc81eaaf9811\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-57fcb7c57d8j8t8"
May 25 11:29:26.016835 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:29:26.016767    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1dfc4e23-6d34-4fe2-af30-dc81eaaf9811-home\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-57fcb7c57d8j8t8\" (UID: \"1dfc4e23-6d34-4fe2-af30-dc81eaaf9811\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-57fcb7c57d8j8t8"
May 25 11:29:26.016835 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:29:26.016815    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-2rc9n\" (UniqueName: \"kubernetes.io/projected/1dfc4e23-6d34-4fe2-af30-dc81eaaf9811-kube-api-access-2rc9n\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-57fcb7c57d8j8t8\" (UID: \"1dfc4e23-6d34-4fe2-af30-dc81eaaf9811\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-57fcb7c57d8j8t8"
May 25 11:29:26.017099 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:29:26.016847    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1dfc4e23-6d34-4fe2-af30-dc81eaaf9811-model-cache\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-57fcb7c57d8j8t8\" (UID: \"1dfc4e23-6d34-4fe2-af30-dc81eaaf9811\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-57fcb7c57d8j8t8"
May 25 11:29:26.017099 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:29:26.017081    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1dfc4e23-6d34-4fe2-af30-dc81eaaf9811-kserve-provision-location\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-57fcb7c57d8j8t8\" (UID: \"1dfc4e23-6d34-4fe2-af30-dc81eaaf9811\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-57fcb7c57d8j8t8"
May 25 11:29:26.017222 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:29:26.017165    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1dfc4e23-6d34-4fe2-af30-dc81eaaf9811-tmp-dir\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-57fcb7c57d8j8t8\" (UID: \"1dfc4e23-6d34-4fe2-af30-dc81eaaf9811\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-57fcb7c57d8j8t8"
May 25 11:29:26.017222 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:29:26.017199    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1dfc4e23-6d34-4fe2-af30-dc81eaaf9811-model-cache\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-57fcb7c57d8j8t8\" (UID: \"1dfc4e23-6d34-4fe2-af30-dc81eaaf9811\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-57fcb7c57d8j8t8"
May 25 11:29:26.017364 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:29:26.017342    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1dfc4e23-6d34-4fe2-af30-dc81eaaf9811-home\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-57fcb7c57d8j8t8\" (UID: \"1dfc4e23-6d34-4fe2-af30-dc81eaaf9811\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-57fcb7c57d8j8t8"
May 25 11:29:26.018955 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:29:26.018935    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1dfc4e23-6d34-4fe2-af30-dc81eaaf9811-dshm\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-57fcb7c57d8j8t8\" (UID: \"1dfc4e23-6d34-4fe2-af30-dc81eaaf9811\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-57fcb7c57d8j8t8"
May 25 11:29:26.019140 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:29:26.019123    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1dfc4e23-6d34-4fe2-af30-dc81eaaf9811-tls-certs\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-57fcb7c57d8j8t8\" (UID: \"1dfc4e23-6d34-4fe2-af30-dc81eaaf9811\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-57fcb7c57d8j8t8"
May 25 11:29:26.024173 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:29:26.024150    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rc9n\" (UniqueName: \"kubernetes.io/projected/1dfc4e23-6d34-4fe2-af30-dc81eaaf9811-kube-api-access-2rc9n\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-57fcb7c57d8j8t8\" (UID: \"1dfc4e23-6d34-4fe2-af30-dc81eaaf9811\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-57fcb7c57d8j8t8"
May 25 11:29:26.199045 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:29:26.199013    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-57fcb7c57d8j8t8"
May 25 11:29:26.329928 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:29:26.329898    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-57fcb7c57d8j8t8"]
May 25 11:29:26.332473 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:29:26.332427    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1dfc4e23_6d34_4fe2_af30_dc81eaaf9811.slice/crio-af544d644c6f6294f9bf303f17ada6e56e8997625c66814b1fba05d61e14a327 WatchSource:0}: Error finding container af544d644c6f6294f9bf303f17ada6e56e8997625c66814b1fba05d61e14a327: Status 404 returned error can't find the container with id af544d644c6f6294f9bf303f17ada6e56e8997625c66814b1fba05d61e14a327
May 25 11:29:26.334365 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:29:26.334335    2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 25 11:29:27.340078 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:29:27.340025    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-57fcb7c57d8j8t8" event={"ID":"1dfc4e23-6d34-4fe2-af30-dc81eaaf9811","Type":"ContainerStarted","Data":"af544d644c6f6294f9bf303f17ada6e56e8997625c66814b1fba05d61e14a327"}
May 25 11:29:30.355230 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:29:30.355193    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-57fcb7c57d8j8t8" event={"ID":"1dfc4e23-6d34-4fe2-af30-dc81eaaf9811","Type":"ContainerStarted","Data":"9bcc5b5ae17d5370d3eb0bd9f28da42cd415d66fec867be75fe679b718aee43b"}
May 25 11:29:39.064607 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:29:39.064566    2576 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-57fcb7c57d8j8t8"]
May 25 11:29:39.065024 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:29:39.064858    2576 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-57fcb7c57d8j8t8" podUID="1dfc4e23-6d34-4fe2-af30-dc81eaaf9811" containerName="storage-initializer" containerID="cri-o://9bcc5b5ae17d5370d3eb0bd9f28da42cd415d66fec867be75fe679b718aee43b" gracePeriod=30
May 25 11:30:09.238862 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:30:09.238836    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_gw-section-name-router-with-gat-2f0a622e-kserve-57fcb7c57d8j8t8_1dfc4e23-6d34-4fe2-af30-dc81eaaf9811/storage-initializer/0.log"
May 25 11:30:09.239268 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:30:09.238919    2576 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-57fcb7c57d8j8t8"
May 25 11:30:09.395268 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:30:09.395176    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1dfc4e23-6d34-4fe2-af30-dc81eaaf9811-tls-certs\") pod \"1dfc4e23-6d34-4fe2-af30-dc81eaaf9811\" (UID: \"1dfc4e23-6d34-4fe2-af30-dc81eaaf9811\") "
May 25 11:30:09.395268 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:30:09.395226    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1dfc4e23-6d34-4fe2-af30-dc81eaaf9811-model-cache\") pod \"1dfc4e23-6d34-4fe2-af30-dc81eaaf9811\" (UID: \"1dfc4e23-6d34-4fe2-af30-dc81eaaf9811\") "
May 25 11:30:09.395268 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:30:09.395259    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1dfc4e23-6d34-4fe2-af30-dc81eaaf9811-dshm\") pod \"1dfc4e23-6d34-4fe2-af30-dc81eaaf9811\" (UID: \"1dfc4e23-6d34-4fe2-af30-dc81eaaf9811\") "
May 25 11:30:09.395564 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:30:09.395322    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2rc9n\" (UniqueName: \"kubernetes.io/projected/1dfc4e23-6d34-4fe2-af30-dc81eaaf9811-kube-api-access-2rc9n\") pod \"1dfc4e23-6d34-4fe2-af30-dc81eaaf9811\" (UID: \"1dfc4e23-6d34-4fe2-af30-dc81eaaf9811\") "
May 25 11:30:09.395564 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:30:09.395351    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1dfc4e23-6d34-4fe2-af30-dc81eaaf9811-home\") pod \"1dfc4e23-6d34-4fe2-af30-dc81eaaf9811\" (UID: \"1dfc4e23-6d34-4fe2-af30-dc81eaaf9811\") "
May 25 11:30:09.395564 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:30:09.395413    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1dfc4e23-6d34-4fe2-af30-dc81eaaf9811-kserve-provision-location\") pod \"1dfc4e23-6d34-4fe2-af30-dc81eaaf9811\" (UID: \"1dfc4e23-6d34-4fe2-af30-dc81eaaf9811\") "
May 25 11:30:09.395564 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:30:09.395442    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1dfc4e23-6d34-4fe2-af30-dc81eaaf9811-tmp-dir\") pod \"1dfc4e23-6d34-4fe2-af30-dc81eaaf9811\" (UID: \"1dfc4e23-6d34-4fe2-af30-dc81eaaf9811\") "
May 25 11:30:09.395564 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:30:09.395551    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1dfc4e23-6d34-4fe2-af30-dc81eaaf9811-model-cache" (OuterVolumeSpecName: "model-cache") pod "1dfc4e23-6d34-4fe2-af30-dc81eaaf9811" (UID: "1dfc4e23-6d34-4fe2-af30-dc81eaaf9811"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:30:09.395804 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:30:09.395732    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1dfc4e23-6d34-4fe2-af30-dc81eaaf9811-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1dfc4e23-6d34-4fe2-af30-dc81eaaf9811" (UID: "1dfc4e23-6d34-4fe2-af30-dc81eaaf9811"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:30:09.395861 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:30:09.395793    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1dfc4e23-6d34-4fe2-af30-dc81eaaf9811-home" (OuterVolumeSpecName: "home") pod "1dfc4e23-6d34-4fe2-af30-dc81eaaf9811" (UID: "1dfc4e23-6d34-4fe2-af30-dc81eaaf9811"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:30:09.395914 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:30:09.395861    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1dfc4e23-6d34-4fe2-af30-dc81eaaf9811-model-cache\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
May 25 11:30:09.395914 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:30:09.395880    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1dfc4e23-6d34-4fe2-af30-dc81eaaf9811-tmp-dir\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
May 25 11:30:09.397389 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:30:09.397361    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1dfc4e23-6d34-4fe2-af30-dc81eaaf9811-dshm" (OuterVolumeSpecName: "dshm") pod "1dfc4e23-6d34-4fe2-af30-dc81eaaf9811" (UID: "1dfc4e23-6d34-4fe2-af30-dc81eaaf9811"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:30:09.397854 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:30:09.397830    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1dfc4e23-6d34-4fe2-af30-dc81eaaf9811-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1dfc4e23-6d34-4fe2-af30-dc81eaaf9811" (UID: "1dfc4e23-6d34-4fe2-af30-dc81eaaf9811"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 11:30:09.397957 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:30:09.397878    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1dfc4e23-6d34-4fe2-af30-dc81eaaf9811-kube-api-access-2rc9n" (OuterVolumeSpecName: "kube-api-access-2rc9n") pod "1dfc4e23-6d34-4fe2-af30-dc81eaaf9811" (UID: "1dfc4e23-6d34-4fe2-af30-dc81eaaf9811"). InnerVolumeSpecName "kube-api-access-2rc9n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 11:30:09.431526 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:30:09.431498    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1dfc4e23-6d34-4fe2-af30-dc81eaaf9811-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "1dfc4e23-6d34-4fe2-af30-dc81eaaf9811" (UID: "1dfc4e23-6d34-4fe2-af30-dc81eaaf9811"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:30:09.487887 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:30:09.487860    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_gw-section-name-router-with-gat-2f0a622e-kserve-57fcb7c57d8j8t8_1dfc4e23-6d34-4fe2-af30-dc81eaaf9811/storage-initializer/0.log"
May 25 11:30:09.488039 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:30:09.487900    2576 generic.go:358] "Generic (PLEG): container finished" podID="1dfc4e23-6d34-4fe2-af30-dc81eaaf9811" containerID="9bcc5b5ae17d5370d3eb0bd9f28da42cd415d66fec867be75fe679b718aee43b" exitCode=137
May 25 11:30:09.488039 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:30:09.487932    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-57fcb7c57d8j8t8" event={"ID":"1dfc4e23-6d34-4fe2-af30-dc81eaaf9811","Type":"ContainerDied","Data":"9bcc5b5ae17d5370d3eb0bd9f28da42cd415d66fec867be75fe679b718aee43b"}
May 25 11:30:09.488039 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:30:09.487973    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-57fcb7c57d8j8t8" event={"ID":"1dfc4e23-6d34-4fe2-af30-dc81eaaf9811","Type":"ContainerDied","Data":"af544d644c6f6294f9bf303f17ada6e56e8997625c66814b1fba05d61e14a327"}
May 25 11:30:09.488039 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:30:09.487977    2576 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-57fcb7c57d8j8t8"
May 25 11:30:09.488039 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:30:09.487988    2576 scope.go:117] "RemoveContainer" containerID="9bcc5b5ae17d5370d3eb0bd9f28da42cd415d66fec867be75fe679b718aee43b"
May 25 11:30:09.498011 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:30:09.497645    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1dfc4e23-6d34-4fe2-af30-dc81eaaf9811-dshm\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
May 25 11:30:09.498011 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:30:09.497681    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2rc9n\" (UniqueName: \"kubernetes.io/projected/1dfc4e23-6d34-4fe2-af30-dc81eaaf9811-kube-api-access-2rc9n\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
May 25 11:30:09.498011 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:30:09.497697    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1dfc4e23-6d34-4fe2-af30-dc81eaaf9811-home\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
May 25 11:30:09.498011 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:30:09.497711    2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1dfc4e23-6d34-4fe2-af30-dc81eaaf9811-kserve-provision-location\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
May 25 11:30:09.498011 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:30:09.497731    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1dfc4e23-6d34-4fe2-af30-dc81eaaf9811-tls-certs\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
May 25 11:30:09.508047 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:30:09.508027    2576 scope.go:117] "RemoveContainer" containerID="9bcc5b5ae17d5370d3eb0bd9f28da42cd415d66fec867be75fe679b718aee43b"
May 25 11:30:09.508346 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:30:09.508317    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9bcc5b5ae17d5370d3eb0bd9f28da42cd415d66fec867be75fe679b718aee43b\": container with ID starting with 9bcc5b5ae17d5370d3eb0bd9f28da42cd415d66fec867be75fe679b718aee43b not found: ID does not exist" containerID="9bcc5b5ae17d5370d3eb0bd9f28da42cd415d66fec867be75fe679b718aee43b"
May 25 11:30:09.508407 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:30:09.508356    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9bcc5b5ae17d5370d3eb0bd9f28da42cd415d66fec867be75fe679b718aee43b"} err="failed to get container status \"9bcc5b5ae17d5370d3eb0bd9f28da42cd415d66fec867be75fe679b718aee43b\": rpc error: code = NotFound desc = could not find container \"9bcc5b5ae17d5370d3eb0bd9f28da42cd415d66fec867be75fe679b718aee43b\": container with ID starting with 9bcc5b5ae17d5370d3eb0bd9f28da42cd415d66fec867be75fe679b718aee43b not found: ID does not exist"
May 25 11:30:09.528129 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:30:09.528094    2576 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-57fcb7c57d8j8t8"]
May 25 11:30:09.531448 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:30:09.531421    2576 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-57fcb7c57d8j8t8"]
May 25 11:30:11.098753 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:30:11.098716    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1dfc4e23-6d34-4fe2-af30-dc81eaaf9811" path="/var/lib/kubelet/pods/1dfc4e23-6d34-4fe2-af30-dc81eaaf9811/volumes"
May 25 11:30:11.178439 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:30:11.178378    2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-84b5f5c9fcbcqsj"]
May 25 11:30:11.178791 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:30:11.178777    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1dfc4e23-6d34-4fe2-af30-dc81eaaf9811" containerName="storage-initializer"
May 25 11:30:11.178843 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:30:11.178793    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="1dfc4e23-6d34-4fe2-af30-dc81eaaf9811" containerName="storage-initializer"
May 25 11:30:11.178878 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:30:11.178851    2576 memory_manager.go:356] "RemoveStaleState removing state" podUID="1dfc4e23-6d34-4fe2-af30-dc81eaaf9811" containerName="storage-initializer"
May 25 11:30:11.184126 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:30:11.184106    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-84b5f5c9fcbcqsj"
May 25 11:30:11.186498 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:30:11.186478    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
May 25 11:30:11.187150 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:30:11.187132    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
May 25 11:30:11.187243 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:30:11.187173    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvad71fa5348b85aebd404221bba611457-kserve-self-signed-certs\""
May 25 11:30:11.187243 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:30:11.187181    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-5cxfn\""
May 25 11:30:11.194404 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:30:11.194379    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-84b5f5c9fcbcqsj"]
May 25 11:30:11.213562 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:30:11.213531    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48f302e5-834a-4850-903f-d05390806c21-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-84b5f5c9fcbcqsj\" (UID: \"48f302e5-834a-4850-903f-d05390806c21\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-84b5f5c9fcbcqsj"
May 25 11:30:11.213726 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:30:11.213570    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/48f302e5-834a-4850-903f-d05390806c21-dshm\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-84b5f5c9fcbcqsj\" (UID: \"48f302e5-834a-4850-903f-d05390806c21\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-84b5f5c9fcbcqsj"
May 25 11:30:11.213726 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:30:11.213652    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/48f302e5-834a-4850-903f-d05390806c21-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-84b5f5c9fcbcqsj\" (UID: \"48f302e5-834a-4850-903f-d05390806c21\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-84b5f5c9fcbcqsj"
May 25 11:30:11.213841 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:30:11.213724    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/48f302e5-834a-4850-903f-d05390806c21-home\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-84b5f5c9fcbcqsj\" (UID: \"48f302e5-834a-4850-903f-d05390806c21\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-84b5f5c9fcbcqsj"
May 25 11:30:11.213841 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:30:11.213761    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48f302e5-834a-4850-903f-d05390806c21-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-84b5f5c9fcbcqsj\" (UID: \"48f302e5-834a-4850-903f-d05390806c21\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-84b5f5c9fcbcqsj"
May 25 11:30:11.213841 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:30:11.213791    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzcs2\" (UniqueName: \"kubernetes.io/projected/48f302e5-834a-4850-903f-d05390806c21-kube-api-access-nzcs2\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-84b5f5c9fcbcqsj\" (UID: \"48f302e5-834a-4850-903f-d05390806c21\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-84b5f5c9fcbcqsj"
May 25 11:30:11.213942 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:30:11.213862    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48f302e5-834a-4850-903f-d05390806c21-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-84b5f5c9fcbcqsj\" (UID: \"48f302e5-834a-4850-903f-d05390806c21\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-84b5f5c9fcbcqsj"
May 25 11:30:11.314341 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:30:11.314299    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48f302e5-834a-4850-903f-d05390806c21-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-84b5f5c9fcbcqsj\" (UID: \"48f302e5-834a-4850-903f-d05390806c21\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-84b5f5c9fcbcqsj"
May 25 11:30:11.314341 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:30:11.314345    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-nzcs2\" (UniqueName: \"kubernetes.io/projected/48f302e5-834a-4850-903f-d05390806c21-kube-api-access-nzcs2\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-84b5f5c9fcbcqsj\" (UID: \"48f302e5-834a-4850-903f-d05390806c21\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-84b5f5c9fcbcqsj"
May 25 11:30:11.314562 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:30:11.314395    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48f302e5-834a-4850-903f-d05390806c21-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-84b5f5c9fcbcqsj\" (UID: \"48f302e5-834a-4850-903f-d05390806c21\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-84b5f5c9fcbcqsj"
May 25 11:30:11.314562 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:30:11.314437    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48f302e5-834a-4850-903f-d05390806c21-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-84b5f5c9fcbcqsj\" (UID: \"48f302e5-834a-4850-903f-d05390806c21\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-84b5f5c9fcbcqsj"
May 25 11:30:11.314562 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:30:11.314457    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/48f302e5-834a-4850-903f-d05390806c21-dshm\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-84b5f5c9fcbcqsj\" (UID: \"48f302e5-834a-4850-903f-d05390806c21\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-84b5f5c9fcbcqsj"
May 25 11:30:11.314562 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:30:11.314489    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/48f302e5-834a-4850-903f-d05390806c21-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-84b5f5c9fcbcqsj\" (UID: \"48f302e5-834a-4850-903f-d05390806c21\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-84b5f5c9fcbcqsj"
May 25 11:30:11.314562 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:30:11.314537    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/48f302e5-834a-4850-903f-d05390806c21-home\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-84b5f5c9fcbcqsj\" (UID: \"48f302e5-834a-4850-903f-d05390806c21\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-84b5f5c9fcbcqsj"
May 25 11:30:11.314953 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:30:11.314920    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48f302e5-834a-4850-903f-d05390806c21-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-84b5f5c9fcbcqsj\" (UID: \"48f302e5-834a-4850-903f-d05390806c21\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-84b5f5c9fcbcqsj"
May 25 11:30:11.315061 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:30:11.314966    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/48f302e5-834a-4850-903f-d05390806c21-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-84b5f5c9fcbcqsj\" (UID: \"48f302e5-834a-4850-903f-d05390806c21\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-84b5f5c9fcbcqsj"
May 25 11:30:11.315061 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:30:11.315016    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48f302e5-834a-4850-903f-d05390806c21-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-84b5f5c9fcbcqsj\" (UID: \"48f302e5-834a-4850-903f-d05390806c21\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-84b5f5c9fcbcqsj"
May 25 11:30:11.315061 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:30:11.315035    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/48f302e5-834a-4850-903f-d05390806c21-home\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-84b5f5c9fcbcqsj\" (UID: \"48f302e5-834a-4850-903f-d05390806c21\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-84b5f5c9fcbcqsj"
May 25 11:30:11.316676 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:30:11.316651    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/48f302e5-834a-4850-903f-d05390806c21-dshm\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-84b5f5c9fcbcqsj\" (UID: \"48f302e5-834a-4850-903f-d05390806c21\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-84b5f5c9fcbcqsj"
May 25 11:30:11.316837 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:30:11.316821    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48f302e5-834a-4850-903f-d05390806c21-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-84b5f5c9fcbcqsj\" (UID: \"48f302e5-834a-4850-903f-d05390806c21\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-84b5f5c9fcbcqsj"
May 25 11:30:11.322454 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:30:11.322433    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzcs2\" (UniqueName: \"kubernetes.io/projected/48f302e5-834a-4850-903f-d05390806c21-kube-api-access-nzcs2\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-84b5f5c9fcbcqsj\" (UID: \"48f302e5-834a-4850-903f-d05390806c21\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-84b5f5c9fcbcqsj"
May 25 11:30:11.495400 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:30:11.495369    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-84b5f5c9fcbcqsj"
May 25 11:30:11.620778 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:30:11.620753    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-84b5f5c9fcbcqsj"]
May 25 11:30:11.623020 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:30:11.622990    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48f302e5_834a_4850_903f_d05390806c21.slice/crio-0a7ea112e3f90cbd50f32a2de7203cd05e40fa04647395b71220c9ac31c42a52 WatchSource:0}: Error finding container 0a7ea112e3f90cbd50f32a2de7203cd05e40fa04647395b71220c9ac31c42a52: Status 404 returned error can't find the container with id 0a7ea112e3f90cbd50f32a2de7203cd05e40fa04647395b71220c9ac31c42a52
May 25 11:30:12.499823 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:30:12.499726    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-84b5f5c9fcbcqsj" event={"ID":"48f302e5-834a-4850-903f-d05390806c21","Type":"ContainerStarted","Data":"49190880a1edccb0049d664e613c86dbeacf5da7f86f7b05cbf6a617ec0fd869"}
May 25 11:30:12.499823 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:30:12.499768    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-84b5f5c9fcbcqsj" event={"ID":"48f302e5-834a-4850-903f-d05390806c21","Type":"ContainerStarted","Data":"0a7ea112e3f90cbd50f32a2de7203cd05e40fa04647395b71220c9ac31c42a52"}
May 25 11:30:17.518631 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:30:17.518593    2576 generic.go:358] "Generic (PLEG): container finished" podID="48f302e5-834a-4850-903f-d05390806c21" containerID="49190880a1edccb0049d664e613c86dbeacf5da7f86f7b05cbf6a617ec0fd869" exitCode=0
May 25 11:30:17.519019 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:30:17.518664    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-84b5f5c9fcbcqsj" event={"ID":"48f302e5-834a-4850-903f-d05390806c21","Type":"ContainerDied","Data":"49190880a1edccb0049d664e613c86dbeacf5da7f86f7b05cbf6a617ec0fd869"}
May 25 11:30:19.528115 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:30:19.528081    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-84b5f5c9fcbcqsj" event={"ID":"48f302e5-834a-4850-903f-d05390806c21","Type":"ContainerStarted","Data":"1b3b117d754d6bf230f1dda6a88922b1dda7ef6942de3ce4dc44c852591f251a"}
May 25 11:30:19.549609 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:30:19.549565    2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-84b5f5c9fcbcqsj" podStartSLOduration=7.328603892 podStartE2EDuration="8.549551914s" podCreationTimestamp="2026-05-25 11:30:11 +0000 UTC" firstStartedPulling="2026-05-25 11:30:17.519838361 +0000 UTC m=+763.037569795" lastFinishedPulling="2026-05-25 11:30:18.740786372 +0000 UTC m=+764.258517817" observedRunningTime="2026-05-25 11:30:19.548756222 +0000 UTC m=+765.066487690" watchObservedRunningTime="2026-05-25 11:30:19.549551914 +0000 UTC m=+765.067283370"
May 25 11:30:21.495905 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:30:21.495862    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-84b5f5c9fcbcqsj"
May 25 11:30:21.495905 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:30:21.495914    2576 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-84b5f5c9fcbcqsj"
May 25 11:30:21.508345 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:30:21.508317    2576 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-84b5f5c9fcbcqsj"
May 25 11:30:21.545555 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:30:21.545532    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-84b5f5c9fcbcqsj"
May 25 11:30:25.102293 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:30:25.102240    2576 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-84b5f5c9fcbcqsj"]
May 25 11:30:25.103048 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:30:25.102494    2576 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-84b5f5c9fcbcqsj" podUID="48f302e5-834a-4850-903f-d05390806c21" containerName="main" containerID="cri-o://1b3b117d754d6bf230f1dda6a88922b1dda7ef6942de3ce4dc44c852591f251a" gracePeriod=30
May 25 11:30:25.385518 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:30:25.385488    2576 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-84b5f5c9fcbcqsj"
May 25 11:30:25.442665 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:30:25.442637    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/48f302e5-834a-4850-903f-d05390806c21-dshm\") pod \"48f302e5-834a-4850-903f-d05390806c21\" (UID: \"48f302e5-834a-4850-903f-d05390806c21\") "
May 25 11:30:25.442822 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:30:25.442671    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/48f302e5-834a-4850-903f-d05390806c21-kserve-provision-location\") pod \"48f302e5-834a-4850-903f-d05390806c21\" (UID: \"48f302e5-834a-4850-903f-d05390806c21\") "
May 25 11:30:25.442822 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:30:25.442711    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48f302e5-834a-4850-903f-d05390806c21-tls-certs\") pod \"48f302e5-834a-4850-903f-d05390806c21\" (UID: \"48f302e5-834a-4850-903f-d05390806c21\") "
May 25 11:30:25.442822 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:30:25.442755    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nzcs2\" (UniqueName: \"kubernetes.io/projected/48f302e5-834a-4850-903f-d05390806c21-kube-api-access-nzcs2\") pod \"48f302e5-834a-4850-903f-d05390806c21\" (UID: \"48f302e5-834a-4850-903f-d05390806c21\") "
May 25 11:30:25.442822 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:30:25.442779    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48f302e5-834a-4850-903f-d05390806c21-model-cache\") pod \"48f302e5-834a-4850-903f-d05390806c21\" (UID: \"48f302e5-834a-4850-903f-d05390806c21\") "
May 25 11:30:25.442822 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:30:25.442799    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48f302e5-834a-4850-903f-d05390806c21-tmp-dir\") pod \"48f302e5-834a-4850-903f-d05390806c21\" (UID: \"48f302e5-834a-4850-903f-d05390806c21\") "
May 25 11:30:25.442822 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:30:25.442820    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/48f302e5-834a-4850-903f-d05390806c21-home\") pod \"48f302e5-834a-4850-903f-d05390806c21\" (UID: \"48f302e5-834a-4850-903f-d05390806c21\") "
May 25 11:30:25.443122 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:30:25.443093    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48f302e5-834a-4850-903f-d05390806c21-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "48f302e5-834a-4850-903f-d05390806c21" (UID: "48f302e5-834a-4850-903f-d05390806c21"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:30:25.443186 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:30:25.443093    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48f302e5-834a-4850-903f-d05390806c21-model-cache" (OuterVolumeSpecName: "model-cache") pod "48f302e5-834a-4850-903f-d05390806c21" (UID: "48f302e5-834a-4850-903f-d05390806c21"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:30:25.443234 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:30:25.443180    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48f302e5-834a-4850-903f-d05390806c21-home" (OuterVolumeSpecName: "home") pod "48f302e5-834a-4850-903f-d05390806c21" (UID: "48f302e5-834a-4850-903f-d05390806c21"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:30:25.445447 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:30:25.445422    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48f302e5-834a-4850-903f-d05390806c21-dshm" (OuterVolumeSpecName: "dshm") pod "48f302e5-834a-4850-903f-d05390806c21" (UID: "48f302e5-834a-4850-903f-d05390806c21"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:30:25.445575 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:30:25.445465    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/48f302e5-834a-4850-903f-d05390806c21-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "48f302e5-834a-4850-903f-d05390806c21" (UID: "48f302e5-834a-4850-903f-d05390806c21"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 11:30:25.445575 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:30:25.445489    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48f302e5-834a-4850-903f-d05390806c21-kube-api-access-nzcs2" (OuterVolumeSpecName: "kube-api-access-nzcs2") pod "48f302e5-834a-4850-903f-d05390806c21" (UID: "48f302e5-834a-4850-903f-d05390806c21"). InnerVolumeSpecName "kube-api-access-nzcs2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 11:30:25.544300 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:30:25.544243    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/48f302e5-834a-4850-903f-d05390806c21-dshm\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
May 25 11:30:25.544300 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:30:25.544301    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48f302e5-834a-4850-903f-d05390806c21-tls-certs\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
May 25 11:30:25.544507 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:30:25.544318    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nzcs2\" (UniqueName: \"kubernetes.io/projected/48f302e5-834a-4850-903f-d05390806c21-kube-api-access-nzcs2\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
May 25 11:30:25.544507 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:30:25.544328    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48f302e5-834a-4850-903f-d05390806c21-model-cache\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
May 25 11:30:25.544507 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:30:25.544337    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48f302e5-834a-4850-903f-d05390806c21-tmp-dir\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
May 25 11:30:25.544507 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:30:25.544344    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/48f302e5-834a-4850-903f-d05390806c21-home\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
May 25 11:30:25.549420 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:30:25.549393    2576 generic.go:358] "Generic (PLEG): container finished" podID="48f302e5-834a-4850-903f-d05390806c21" containerID="1b3b117d754d6bf230f1dda6a88922b1dda7ef6942de3ce4dc44c852591f251a" exitCode=0
May 25 11:30:25.549544 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:30:25.549462    2576 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-84b5f5c9fcbcqsj"
May 25 11:30:25.549544 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:30:25.549462    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-84b5f5c9fcbcqsj" event={"ID":"48f302e5-834a-4850-903f-d05390806c21","Type":"ContainerDied","Data":"1b3b117d754d6bf230f1dda6a88922b1dda7ef6942de3ce4dc44c852591f251a"}
May 25 11:30:25.549621 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:30:25.549563    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-84b5f5c9fcbcqsj" event={"ID":"48f302e5-834a-4850-903f-d05390806c21","Type":"ContainerDied","Data":"0a7ea112e3f90cbd50f32a2de7203cd05e40fa04647395b71220c9ac31c42a52"}
May 25 11:30:25.549621 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:30:25.549578    2576 scope.go:117] "RemoveContainer" containerID="1b3b117d754d6bf230f1dda6a88922b1dda7ef6942de3ce4dc44c852591f251a"
May 25 11:30:25.557538 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:30:25.557516    2576 scope.go:117] "RemoveContainer" containerID="49190880a1edccb0049d664e613c86dbeacf5da7f86f7b05cbf6a617ec0fd869"
May 25 11:30:25.622221 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:30:25.622201    2576 scope.go:117] "RemoveContainer" containerID="1b3b117d754d6bf230f1dda6a88922b1dda7ef6942de3ce4dc44c852591f251a"
May 25 11:30:25.622579 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:30:25.622556    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b3b117d754d6bf230f1dda6a88922b1dda7ef6942de3ce4dc44c852591f251a\": container with ID starting with 1b3b117d754d6bf230f1dda6a88922b1dda7ef6942de3ce4dc44c852591f251a not found: ID does not exist" containerID="1b3b117d754d6bf230f1dda6a88922b1dda7ef6942de3ce4dc44c852591f251a"
May 25 11:30:25.622656 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:30:25.622588    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b3b117d754d6bf230f1dda6a88922b1dda7ef6942de3ce4dc44c852591f251a"} err="failed to get container status \"1b3b117d754d6bf230f1dda6a88922b1dda7ef6942de3ce4dc44c852591f251a\": rpc error: code = NotFound desc = could not find container \"1b3b117d754d6bf230f1dda6a88922b1dda7ef6942de3ce4dc44c852591f251a\": container with ID starting with 1b3b117d754d6bf230f1dda6a88922b1dda7ef6942de3ce4dc44c852591f251a not found: ID does not exist"
May 25 11:30:25.622656 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:30:25.622609    2576 scope.go:117] "RemoveContainer" containerID="49190880a1edccb0049d664e613c86dbeacf5da7f86f7b05cbf6a617ec0fd869"
May 25 11:30:25.622829 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:30:25.622814    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49190880a1edccb0049d664e613c86dbeacf5da7f86f7b05cbf6a617ec0fd869\": container with ID starting with 49190880a1edccb0049d664e613c86dbeacf5da7f86f7b05cbf6a617ec0fd869 not found: ID does not exist" containerID="49190880a1edccb0049d664e613c86dbeacf5da7f86f7b05cbf6a617ec0fd869"
May 25 11:30:25.622872 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:30:25.622834    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49190880a1edccb0049d664e613c86dbeacf5da7f86f7b05cbf6a617ec0fd869"} err="failed to get container status \"49190880a1edccb0049d664e613c86dbeacf5da7f86f7b05cbf6a617ec0fd869\": rpc error: code = NotFound desc = could not find container \"49190880a1edccb0049d664e613c86dbeacf5da7f86f7b05cbf6a617ec0fd869\": container with ID starting with 49190880a1edccb0049d664e613c86dbeacf5da7f86f7b05cbf6a617ec0fd869 not found: ID does not exist"
May 25 11:30:26.002568 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:30:26.002500    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48f302e5-834a-4850-903f-d05390806c21-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "48f302e5-834a-4850-903f-d05390806c21" (UID: "48f302e5-834a-4850-903f-d05390806c21"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:30:26.048926 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:30:26.048886    2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/48f302e5-834a-4850-903f-d05390806c21-kserve-provision-location\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
May 25 11:30:26.171272 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:30:26.171244    2576 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-84b5f5c9fcbcqsj"]
May 25 11:30:26.176103 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:30:26.176080    2576 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-84b5f5c9fcbcqsj"]
May 25 11:30:27.098303 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:30:27.098246    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="48f302e5-834a-4850-903f-d05390806c21" path="/var/lib/kubelet/pods/48f302e5-834a-4850-903f-d05390806c21/volumes"
May 25 11:30:38.620491 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:30:38.620457    2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-6cfcc85fbfgwqkr"]
May 25 11:30:38.620899 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:30:38.620875    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="48f302e5-834a-4850-903f-d05390806c21" containerName="storage-initializer"
May 25 11:30:38.620899 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:30:38.620894    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="48f302e5-834a-4850-903f-d05390806c21" containerName="storage-initializer"
May 25 11:30:38.621005 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:30:38.620908    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="48f302e5-834a-4850-903f-d05390806c21" containerName="main"
May 25 11:30:38.621005 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:30:38.620918    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="48f302e5-834a-4850-903f-d05390806c21" containerName="main"
May 25 11:30:38.621067 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:30:38.621028    2576 memory_manager.go:356] "RemoveStaleState removing state" podUID="48f302e5-834a-4850-903f-d05390806c21" containerName="main"
May 25 11:30:38.625315 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:30:38.625294    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-6cfcc85fbfgwqkr"
May 25 11:30:38.630663 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:30:38.630634    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
May 25 11:30:38.630663 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:30:38.630650    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvdde380eaa9fe1facad32d45131f9e34d-kserve-self-signed-certs\""
May 25 11:30:38.636816 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:30:38.636797    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
May 25 11:30:38.639366 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:30:38.639351    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-5cxfn\""
May 25 11:30:38.654766 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:30:38.654737    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-6cfcc85fbfgwqkr"]
May 25 11:30:38.763955 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:30:38.763911    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54f7c330-1562-4ba6-8cb5-9e509386ae04-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-6cfcc85fbfgwqkr\" (UID: \"54f7c330-1562-4ba6-8cb5-9e509386ae04\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-6cfcc85fbfgwqkr"
May 25 11:30:38.763955 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:30:38.763956    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/54f7c330-1562-4ba6-8cb5-9e509386ae04-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-6cfcc85fbfgwqkr\" (UID: \"54f7c330-1562-4ba6-8cb5-9e509386ae04\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-6cfcc85fbfgwqkr"
May 25 11:30:38.764227 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:30:38.764085    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54f7c330-1562-4ba6-8cb5-9e509386ae04-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-6cfcc85fbfgwqkr\" (UID: \"54f7c330-1562-4ba6-8cb5-9e509386ae04\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-6cfcc85fbfgwqkr"
May 25 11:30:38.764227 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:30:38.764155    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/54f7c330-1562-4ba6-8cb5-9e509386ae04-home\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-6cfcc85fbfgwqkr\" (UID: \"54f7c330-1562-4ba6-8cb5-9e509386ae04\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-6cfcc85fbfgwqkr"
May 25 11:30:38.764227 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:30:38.764188    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7wvr\" (UniqueName: \"kubernetes.io/projected/54f7c330-1562-4ba6-8cb5-9e509386ae04-kube-api-access-t7wvr\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-6cfcc85fbfgwqkr\" (UID: \"54f7c330-1562-4ba6-8cb5-9e509386ae04\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-6cfcc85fbfgwqkr"
May 25 11:30:38.764227 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:30:38.764223    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54f7c330-1562-4ba6-8cb5-9e509386ae04-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-6cfcc85fbfgwqkr\" (UID: \"54f7c330-1562-4ba6-8cb5-9e509386ae04\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-6cfcc85fbfgwqkr"
May 25 11:30:38.764436 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:30:38.764328    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/54f7c330-1562-4ba6-8cb5-9e509386ae04-dshm\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-6cfcc85fbfgwqkr\" (UID: \"54f7c330-1562-4ba6-8cb5-9e509386ae04\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-6cfcc85fbfgwqkr"
May 25 11:30:38.864937 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:30:38.864899    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54f7c330-1562-4ba6-8cb5-9e509386ae04-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-6cfcc85fbfgwqkr\" (UID: \"54f7c330-1562-4ba6-8cb5-9e509386ae04\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-6cfcc85fbfgwqkr"
May 25 11:30:38.865123 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:30:38.864949    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/54f7c330-1562-4ba6-8cb5-9e509386ae04-home\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-6cfcc85fbfgwqkr\" (UID: \"54f7c330-1562-4ba6-8cb5-9e509386ae04\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-6cfcc85fbfgwqkr"
May 25 11:30:38.865123 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:30:38.864970    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-t7wvr\" (UniqueName: \"kubernetes.io/projected/54f7c330-1562-4ba6-8cb5-9e509386ae04-kube-api-access-t7wvr\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-6cfcc85fbfgwqkr\" (UID: \"54f7c330-1562-4ba6-8cb5-9e509386ae04\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-6cfcc85fbfgwqkr"
May 25 11:30:38.865123 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:30:38.864993    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54f7c330-1562-4ba6-8cb5-9e509386ae04-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-6cfcc85fbfgwqkr\" (UID: \"54f7c330-1562-4ba6-8cb5-9e509386ae04\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-6cfcc85fbfgwqkr"
May 25 11:30:38.865123 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:30:38.865021    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/54f7c330-1562-4ba6-8cb5-9e509386ae04-dshm\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-6cfcc85fbfgwqkr\" (UID: \"54f7c330-1562-4ba6-8cb5-9e509386ae04\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-6cfcc85fbfgwqkr"
May 25 11:30:38.865123 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:30:38.865055    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54f7c330-1562-4ba6-8cb5-9e509386ae04-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-6cfcc85fbfgwqkr\" (UID: \"54f7c330-1562-4ba6-8cb5-9e509386ae04\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-6cfcc85fbfgwqkr"
May 25 11:30:38.865123 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:30:38.865072    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/54f7c330-1562-4ba6-8cb5-9e509386ae04-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-6cfcc85fbfgwqkr\" (UID: \"54f7c330-1562-4ba6-8cb5-9e509386ae04\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-6cfcc85fbfgwqkr"
May 25 11:30:38.865448 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:30:38.865425    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/54f7c330-1562-4ba6-8cb5-9e509386ae04-home\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-6cfcc85fbfgwqkr\" (UID: \"54f7c330-1562-4ba6-8cb5-9e509386ae04\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-6cfcc85fbfgwqkr"
May 25 11:30:38.865487 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:30:38.865462    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54f7c330-1562-4ba6-8cb5-9e509386ae04-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-6cfcc85fbfgwqkr\" (UID: \"54f7c330-1562-4ba6-8cb5-9e509386ae04\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-6cfcc85fbfgwqkr"
May 25 11:30:38.865521 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:30:38.865480    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54f7c330-1562-4ba6-8cb5-9e509386ae04-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-6cfcc85fbfgwqkr\" (UID: \"54f7c330-1562-4ba6-8cb5-9e509386ae04\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-6cfcc85fbfgwqkr"
May 25 11:30:38.865554 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:30:38.865525    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/54f7c330-1562-4ba6-8cb5-9e509386ae04-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-6cfcc85fbfgwqkr\" (UID: \"54f7c330-1562-4ba6-8cb5-9e509386ae04\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-6cfcc85fbfgwqkr"
May 25 11:30:38.867465 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:30:38.867438    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/54f7c330-1562-4ba6-8cb5-9e509386ae04-dshm\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-6cfcc85fbfgwqkr\" (UID: \"54f7c330-1562-4ba6-8cb5-9e509386ae04\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-6cfcc85fbfgwqkr"
May 25 11:30:38.867711 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:30:38.867688    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54f7c330-1562-4ba6-8cb5-9e509386ae04-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-6cfcc85fbfgwqkr\" (UID: \"54f7c330-1562-4ba6-8cb5-9e509386ae04\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-6cfcc85fbfgwqkr"
May 25 11:30:38.875864 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:30:38.875791    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7wvr\" (UniqueName: \"kubernetes.io/projected/54f7c330-1562-4ba6-8cb5-9e509386ae04-kube-api-access-t7wvr\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-6cfcc85fbfgwqkr\" (UID: \"54f7c330-1562-4ba6-8cb5-9e509386ae04\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-6cfcc85fbfgwqkr"
May 25 11:30:38.935738 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:30:38.935702    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-6cfcc85fbfgwqkr"
May 25 11:30:39.078408 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:30:39.078377    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-6cfcc85fbfgwqkr"]
May 25 11:30:39.081072 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:30:39.081043    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54f7c330_1562_4ba6_8cb5_9e509386ae04.slice/crio-c6f64eb3bdd78283ad98b999b6fd73e8e3bf9f7c9bfca0890fecb20a451c95b8 WatchSource:0}: Error finding container c6f64eb3bdd78283ad98b999b6fd73e8e3bf9f7c9bfca0890fecb20a451c95b8: Status 404 returned error can't find the container with id c6f64eb3bdd78283ad98b999b6fd73e8e3bf9f7c9bfca0890fecb20a451c95b8
May 25 11:30:39.601625 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:30:39.601582    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-6cfcc85fbfgwqkr" event={"ID":"54f7c330-1562-4ba6-8cb5-9e509386ae04","Type":"ContainerStarted","Data":"d2151d6147dee1fe766ac51b9eb8d63e439da83045377e84f994a7bd937f0032"}
May 25 11:30:39.601788 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:30:39.601634    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-6cfcc85fbfgwqkr" event={"ID":"54f7c330-1562-4ba6-8cb5-9e509386ae04","Type":"ContainerStarted","Data":"c6f64eb3bdd78283ad98b999b6fd73e8e3bf9f7c9bfca0890fecb20a451c95b8"}
May 25 11:30:44.620529 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:30:44.620493    2576 generic.go:358] "Generic (PLEG): container finished" podID="54f7c330-1562-4ba6-8cb5-9e509386ae04" containerID="d2151d6147dee1fe766ac51b9eb8d63e439da83045377e84f994a7bd937f0032" exitCode=0
May 25 11:30:44.620997 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:30:44.620566    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-6cfcc85fbfgwqkr" event={"ID":"54f7c330-1562-4ba6-8cb5-9e509386ae04","Type":"ContainerDied","Data":"d2151d6147dee1fe766ac51b9eb8d63e439da83045377e84f994a7bd937f0032"}
May 25 11:31:15.753824 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:31:15.753776    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-6cfcc85fbfgwqkr" event={"ID":"54f7c330-1562-4ba6-8cb5-9e509386ae04","Type":"ContainerStarted","Data":"511090d28b5709858e97dbc51b7afdd01e68fe711f68f289f0594478d8985456"}
May 25 11:31:15.776444 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:31:15.776388    2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-6cfcc85fbfgwqkr" podStartSLOduration=7.401857894 podStartE2EDuration="37.776369732s" podCreationTimestamp="2026-05-25 11:30:38 +0000 UTC" firstStartedPulling="2026-05-25 11:30:44.621678425 +0000 UTC m=+790.139409863" lastFinishedPulling="2026-05-25 11:31:14.99619025 +0000 UTC m=+820.513921701" observedRunningTime="2026-05-25 11:31:15.774809335 +0000 UTC m=+821.292540791" watchObservedRunningTime="2026-05-25 11:31:15.776369732 +0000 UTC m=+821.294101190"
May 25 11:31:18.936720 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:31:18.936684    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-6cfcc85fbfgwqkr"
May 25 11:31:18.937140 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:31:18.936772    2576 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-6cfcc85fbfgwqkr"
May 25 11:31:18.938292 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:31:18.938253    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-6cfcc85fbfgwqkr" podUID="54f7c330-1562-4ba6-8cb5-9e509386ae04" containerName="main" probeResult="failure" output="Get \"https://10.133.0.31:8000/health\": dial tcp 10.133.0.31:8000: connect: connection refused"
May 25 11:31:28.936732 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:31:28.936685    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-6cfcc85fbfgwqkr" podUID="54f7c330-1562-4ba6-8cb5-9e509386ae04" containerName="main" probeResult="failure" output="Get \"https://10.133.0.31:8000/health\": dial tcp 10.133.0.31:8000: connect: connection refused"
May 25 11:31:38.936783 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:31:38.936731    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-6cfcc85fbfgwqkr" podUID="54f7c330-1562-4ba6-8cb5-9e509386ae04" containerName="main" probeResult="failure" output="Get \"https://10.133.0.31:8000/health\": dial tcp 10.133.0.31:8000: connect: connection refused"
May 25 11:31:48.936542 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:31:48.936493    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-6cfcc85fbfgwqkr" podUID="54f7c330-1562-4ba6-8cb5-9e509386ae04" containerName="main" probeResult="failure" output="Get \"https://10.133.0.31:8000/health\": dial tcp 10.133.0.31:8000: connect: connection refused"
May 25 11:31:58.936341 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:31:58.936292    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-6cfcc85fbfgwqkr" podUID="54f7c330-1562-4ba6-8cb5-9e509386ae04" containerName="main" probeResult="failure" output="Get \"https://10.133.0.31:8000/health\": dial tcp 10.133.0.31:8000: connect: connection refused"
May 25 11:32:08.936770 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:32:08.936719    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-6cfcc85fbfgwqkr" podUID="54f7c330-1562-4ba6-8cb5-9e509386ae04" containerName="main" probeResult="failure" output="Get \"https://10.133.0.31:8000/health\": dial tcp 10.133.0.31:8000: connect: connection refused"
May 25 11:32:18.936826 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:32:18.936731    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-6cfcc85fbfgwqkr" podUID="54f7c330-1562-4ba6-8cb5-9e509386ae04" containerName="main" probeResult="failure" output="Get \"https://10.133.0.31:8000/health\": dial tcp 10.133.0.31:8000: connect: connection refused"
May 25 11:32:28.936433 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:32:28.936383    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-6cfcc85fbfgwqkr" podUID="54f7c330-1562-4ba6-8cb5-9e509386ae04" containerName="main" probeResult="failure" output="Get \"https://10.133.0.31:8000/health\": dial tcp 10.133.0.31:8000: connect: connection refused"
May 25 11:32:35.061924 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:32:35.061892    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h6b5t_12b45353-8444-41fc-a8ed-fb691764eb63/ovn-acl-logging/0.log"
May 25 11:32:35.062501 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:32:35.062481    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h6b5t_12b45353-8444-41fc-a8ed-fb691764eb63/ovn-acl-logging/0.log"
May 25 11:32:38.936579 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:32:38.936531    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-6cfcc85fbfgwqkr" podUID="54f7c330-1562-4ba6-8cb5-9e509386ae04" containerName="main" probeResult="failure" output="Get \"https://10.133.0.31:8000/health\": dial tcp 10.133.0.31:8000: connect: connection refused"
May 25 11:32:48.946684 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:32:48.946649    2576 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-6cfcc85fbfgwqkr"
May 25 11:32:48.954828 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:32:48.954799    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-6cfcc85fbfgwqkr"
May 25 11:32:55.492227 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:32:55.492188    2576 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-6cfcc85fbfgwqkr"]
May 25 11:32:55.492715 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:32:55.492482    2576 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-6cfcc85fbfgwqkr" podUID="54f7c330-1562-4ba6-8cb5-9e509386ae04" containerName="main" containerID="cri-o://511090d28b5709858e97dbc51b7afdd01e68fe711f68f289f0594478d8985456" gracePeriod=30
May 25 11:33:05.477191 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:33:05.477154    2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-6c579d5bdf-fh4bf"]
May 25 11:33:05.481039 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:33:05.481019    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-6c579d5bdf-fh4bf"
May 25 11:33:05.483399 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:33:05.483378    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"custom-route-timeout-test-kserve-self-signed-certs\""
May 25 11:33:05.494315 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:33:05.494265    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-6c579d5bdf-fh4bf"]
May 25 11:33:05.618582 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:33:05.618547    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/230c99ee-7273-42d7-be1d-a075fbbca797-dshm\") pod \"custom-route-timeout-test-kserve-6c579d5bdf-fh4bf\" (UID: \"230c99ee-7273-42d7-be1d-a075fbbca797\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-6c579d5bdf-fh4bf"
May 25 11:33:05.618778 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:33:05.618626    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/230c99ee-7273-42d7-be1d-a075fbbca797-model-cache\") pod \"custom-route-timeout-test-kserve-6c579d5bdf-fh4bf\" (UID: \"230c99ee-7273-42d7-be1d-a075fbbca797\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-6c579d5bdf-fh4bf"
May 25 11:33:05.618778 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:33:05.618676    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/230c99ee-7273-42d7-be1d-a075fbbca797-tmp-dir\") pod \"custom-route-timeout-test-kserve-6c579d5bdf-fh4bf\" (UID: \"230c99ee-7273-42d7-be1d-a075fbbca797\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-6c579d5bdf-fh4bf"
May 25 11:33:05.618778 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:33:05.618738    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjrx4\" (UniqueName: \"kubernetes.io/projected/230c99ee-7273-42d7-be1d-a075fbbca797-kube-api-access-hjrx4\") pod \"custom-route-timeout-test-kserve-6c579d5bdf-fh4bf\" (UID: \"230c99ee-7273-42d7-be1d-a075fbbca797\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-6c579d5bdf-fh4bf"
May 25 11:33:05.618917 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:33:05.618798    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/230c99ee-7273-42d7-be1d-a075fbbca797-home\") pod \"custom-route-timeout-test-kserve-6c579d5bdf-fh4bf\" (UID: \"230c99ee-7273-42d7-be1d-a075fbbca797\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-6c579d5bdf-fh4bf"
May 25 11:33:05.618917 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:33:05.618827    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/230c99ee-7273-42d7-be1d-a075fbbca797-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-6c579d5bdf-fh4bf\" (UID: \"230c99ee-7273-42d7-be1d-a075fbbca797\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-6c579d5bdf-fh4bf"
May 25 11:33:05.618917 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:33:05.618870    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/230c99ee-7273-42d7-be1d-a075fbbca797-tls-certs\") pod \"custom-route-timeout-test-kserve-6c579d5bdf-fh4bf\" (UID: \"230c99ee-7273-42d7-be1d-a075fbbca797\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-6c579d5bdf-fh4bf"
May 25 11:33:05.719762 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:33:05.719718    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/230c99ee-7273-42d7-be1d-a075fbbca797-home\") pod \"custom-route-timeout-test-kserve-6c579d5bdf-fh4bf\" (UID: \"230c99ee-7273-42d7-be1d-a075fbbca797\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-6c579d5bdf-fh4bf"
May 25 11:33:05.719762 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:33:05.719765    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/230c99ee-7273-42d7-be1d-a075fbbca797-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-6c579d5bdf-fh4bf\" (UID: \"230c99ee-7273-42d7-be1d-a075fbbca797\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-6c579d5bdf-fh4bf"
May 25 11:33:05.720012 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:33:05.719796    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/230c99ee-7273-42d7-be1d-a075fbbca797-tls-certs\") pod \"custom-route-timeout-test-kserve-6c579d5bdf-fh4bf\" (UID: \"230c99ee-7273-42d7-be1d-a075fbbca797\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-6c579d5bdf-fh4bf"
May 25 11:33:05.720012 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:33:05.719847    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/230c99ee-7273-42d7-be1d-a075fbbca797-dshm\") pod \"custom-route-timeout-test-kserve-6c579d5bdf-fh4bf\" (UID: \"230c99ee-7273-42d7-be1d-a075fbbca797\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-6c579d5bdf-fh4bf"
May 25 11:33:05.720012 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:33:05.719905    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/230c99ee-7273-42d7-be1d-a075fbbca797-model-cache\") pod \"custom-route-timeout-test-kserve-6c579d5bdf-fh4bf\" (UID: \"230c99ee-7273-42d7-be1d-a075fbbca797\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-6c579d5bdf-fh4bf"
May 25 11:33:05.720012 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:33:05.719953    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/230c99ee-7273-42d7-be1d-a075fbbca797-tmp-dir\") pod \"custom-route-timeout-test-kserve-6c579d5bdf-fh4bf\" (UID: \"230c99ee-7273-42d7-be1d-a075fbbca797\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-6c579d5bdf-fh4bf"
May 25 11:33:05.720012 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:33:05.719989    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-hjrx4\" (UniqueName: \"kubernetes.io/projected/230c99ee-7273-42d7-be1d-a075fbbca797-kube-api-access-hjrx4\") pod \"custom-route-timeout-test-kserve-6c579d5bdf-fh4bf\" (UID: \"230c99ee-7273-42d7-be1d-a075fbbca797\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-6c579d5bdf-fh4bf"
May 25 11:33:05.720267 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:33:05.720148    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/230c99ee-7273-42d7-be1d-a075fbbca797-home\") pod \"custom-route-timeout-test-kserve-6c579d5bdf-fh4bf\" (UID: \"230c99ee-7273-42d7-be1d-a075fbbca797\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-6c579d5bdf-fh4bf"
May 25 11:33:05.720267 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:33:05.720184    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/230c99ee-7273-42d7-be1d-a075fbbca797-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-6c579d5bdf-fh4bf\" (UID: \"230c99ee-7273-42d7-be1d-a075fbbca797\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-6c579d5bdf-fh4bf"
May 25 11:33:05.720267 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:33:05.720260    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/230c99ee-7273-42d7-be1d-a075fbbca797-model-cache\") pod \"custom-route-timeout-test-kserve-6c579d5bdf-fh4bf\" (UID: \"230c99ee-7273-42d7-be1d-a075fbbca797\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-6c579d5bdf-fh4bf"
May 25 11:33:05.720404 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:33:05.720359    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/230c99ee-7273-42d7-be1d-a075fbbca797-tmp-dir\") pod \"custom-route-timeout-test-kserve-6c579d5bdf-fh4bf\" (UID: \"230c99ee-7273-42d7-be1d-a075fbbca797\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-6c579d5bdf-fh4bf"
May 25 11:33:05.722092 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:33:05.722070    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/230c99ee-7273-42d7-be1d-a075fbbca797-dshm\") pod \"custom-route-timeout-test-kserve-6c579d5bdf-fh4bf\" (UID: \"230c99ee-7273-42d7-be1d-a075fbbca797\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-6c579d5bdf-fh4bf"
May 25 11:33:05.722447 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:33:05.722427    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/230c99ee-7273-42d7-be1d-a075fbbca797-tls-certs\") pod \"custom-route-timeout-test-kserve-6c579d5bdf-fh4bf\" (UID: \"230c99ee-7273-42d7-be1d-a075fbbca797\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-6c579d5bdf-fh4bf"
May 25 11:33:05.731004 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:33:05.730939    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjrx4\" (UniqueName: \"kubernetes.io/projected/230c99ee-7273-42d7-be1d-a075fbbca797-kube-api-access-hjrx4\") pod \"custom-route-timeout-test-kserve-6c579d5bdf-fh4bf\" (UID: \"230c99ee-7273-42d7-be1d-a075fbbca797\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-6c579d5bdf-fh4bf"
May 25 11:33:05.791327 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:33:05.791261    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-6c579d5bdf-fh4bf"
May 25 11:33:05.924679 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:33:05.924653    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-6c579d5bdf-fh4bf"]
May 25 11:33:05.926841 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:33:05.926808    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod230c99ee_7273_42d7_be1d_a075fbbca797.slice/crio-23d72c1adf82d036c21849a7cf084c0a4d838246d06e7e7b40af489aa02ae9b6 WatchSource:0}: Error finding container 23d72c1adf82d036c21849a7cf084c0a4d838246d06e7e7b40af489aa02ae9b6: Status 404 returned error can't find the container with id 23d72c1adf82d036c21849a7cf084c0a4d838246d06e7e7b40af489aa02ae9b6
May 25 11:33:06.153243 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:33:06.153201    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-6c579d5bdf-fh4bf" event={"ID":"230c99ee-7273-42d7-be1d-a075fbbca797","Type":"ContainerStarted","Data":"f36bf81bbf28e340e41b2d0f21160e8fdbd5378ec277e8db4b49faf273189cc1"}
May 25 11:33:06.153243 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:33:06.153247    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-6c579d5bdf-fh4bf" event={"ID":"230c99ee-7273-42d7-be1d-a075fbbca797","Type":"ContainerStarted","Data":"23d72c1adf82d036c21849a7cf084c0a4d838246d06e7e7b40af489aa02ae9b6"}
May 25 11:33:25.758963 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:33:25.758898    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-6cfcc85fbfgwqkr_54f7c330-1562-4ba6-8cb5-9e509386ae04/main/0.log"
May 25 11:33:25.759351 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:33:25.759245    2576 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-6cfcc85fbfgwqkr"
May 25 11:33:25.800688 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:33:25.800651    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54f7c330-1562-4ba6-8cb5-9e509386ae04-tls-certs\") pod \"54f7c330-1562-4ba6-8cb5-9e509386ae04\" (UID: \"54f7c330-1562-4ba6-8cb5-9e509386ae04\") "
May 25 11:33:25.800887 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:33:25.800715    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54f7c330-1562-4ba6-8cb5-9e509386ae04-tmp-dir\") pod \"54f7c330-1562-4ba6-8cb5-9e509386ae04\" (UID: \"54f7c330-1562-4ba6-8cb5-9e509386ae04\") "
May 25 11:33:25.800887 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:33:25.800777    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/54f7c330-1562-4ba6-8cb5-9e509386ae04-home\") pod \"54f7c330-1562-4ba6-8cb5-9e509386ae04\" (UID: \"54f7c330-1562-4ba6-8cb5-9e509386ae04\") "
May 25 11:33:25.800887 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:33:25.800800    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t7wvr\" (UniqueName: \"kubernetes.io/projected/54f7c330-1562-4ba6-8cb5-9e509386ae04-kube-api-access-t7wvr\") pod \"54f7c330-1562-4ba6-8cb5-9e509386ae04\" (UID: \"54f7c330-1562-4ba6-8cb5-9e509386ae04\") "
May 25 11:33:25.800887 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:33:25.800822    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/54f7c330-1562-4ba6-8cb5-9e509386ae04-dshm\") pod \"54f7c330-1562-4ba6-8cb5-9e509386ae04\" (UID: \"54f7c330-1562-4ba6-8cb5-9e509386ae04\") "
May 25 11:33:25.800887 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:33:25.800853    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/54f7c330-1562-4ba6-8cb5-9e509386ae04-kserve-provision-location\") pod \"54f7c330-1562-4ba6-8cb5-9e509386ae04\" (UID: \"54f7c330-1562-4ba6-8cb5-9e509386ae04\") "
May 25 11:33:25.800887 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:33:25.800883    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54f7c330-1562-4ba6-8cb5-9e509386ae04-model-cache\") pod \"54f7c330-1562-4ba6-8cb5-9e509386ae04\" (UID: \"54f7c330-1562-4ba6-8cb5-9e509386ae04\") "
May 25 11:33:25.801315 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:33:25.801268    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54f7c330-1562-4ba6-8cb5-9e509386ae04-model-cache" (OuterVolumeSpecName: "model-cache") pod "54f7c330-1562-4ba6-8cb5-9e509386ae04" (UID: "54f7c330-1562-4ba6-8cb5-9e509386ae04"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:33:25.801518 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:33:25.801497    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54f7c330-1562-4ba6-8cb5-9e509386ae04-home" (OuterVolumeSpecName: "home") pod "54f7c330-1562-4ba6-8cb5-9e509386ae04" (UID: "54f7c330-1562-4ba6-8cb5-9e509386ae04"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:33:25.803337 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:33:25.803306    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54f7c330-1562-4ba6-8cb5-9e509386ae04-kube-api-access-t7wvr" (OuterVolumeSpecName: "kube-api-access-t7wvr") pod "54f7c330-1562-4ba6-8cb5-9e509386ae04" (UID: "54f7c330-1562-4ba6-8cb5-9e509386ae04"). InnerVolumeSpecName "kube-api-access-t7wvr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 11:33:25.803449 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:33:25.803371    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54f7c330-1562-4ba6-8cb5-9e509386ae04-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "54f7c330-1562-4ba6-8cb5-9e509386ae04" (UID: "54f7c330-1562-4ba6-8cb5-9e509386ae04"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 11:33:25.803746 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:33:25.803724    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54f7c330-1562-4ba6-8cb5-9e509386ae04-dshm" (OuterVolumeSpecName: "dshm") pod "54f7c330-1562-4ba6-8cb5-9e509386ae04" (UID: "54f7c330-1562-4ba6-8cb5-9e509386ae04"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:33:25.812676 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:33:25.812642    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54f7c330-1562-4ba6-8cb5-9e509386ae04-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "54f7c330-1562-4ba6-8cb5-9e509386ae04" (UID: "54f7c330-1562-4ba6-8cb5-9e509386ae04"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:33:25.858235 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:33:25.858191    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54f7c330-1562-4ba6-8cb5-9e509386ae04-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "54f7c330-1562-4ba6-8cb5-9e509386ae04" (UID: "54f7c330-1562-4ba6-8cb5-9e509386ae04"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:33:25.902035 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:33:25.902000    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/54f7c330-1562-4ba6-8cb5-9e509386ae04-home\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
May 25 11:33:25.902035 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:33:25.902035    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t7wvr\" (UniqueName: \"kubernetes.io/projected/54f7c330-1562-4ba6-8cb5-9e509386ae04-kube-api-access-t7wvr\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
May 25 11:33:25.902187 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:33:25.902047    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/54f7c330-1562-4ba6-8cb5-9e509386ae04-dshm\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
May 25 11:33:25.902187 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:33:25.902056    2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/54f7c330-1562-4ba6-8cb5-9e509386ae04-kserve-provision-location\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
May 25 11:33:25.902187 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:33:25.902080    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54f7c330-1562-4ba6-8cb5-9e509386ae04-model-cache\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
May 25 11:33:25.902187 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:33:25.902091    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54f7c330-1562-4ba6-8cb5-9e509386ae04-tls-certs\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
May 25 11:33:25.902187 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:33:25.902099    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54f7c330-1562-4ba6-8cb5-9e509386ae04-tmp-dir\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
May 25 11:33:26.227009 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:33:26.226981    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-6cfcc85fbfgwqkr_54f7c330-1562-4ba6-8cb5-9e509386ae04/main/0.log"
May 25 11:33:26.227354 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:33:26.227329    2576 generic.go:358] "Generic (PLEG): container finished" podID="54f7c330-1562-4ba6-8cb5-9e509386ae04" containerID="511090d28b5709858e97dbc51b7afdd01e68fe711f68f289f0594478d8985456" exitCode=137
May 25 11:33:26.227445 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:33:26.227410    2576 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-6cfcc85fbfgwqkr"
May 25 11:33:26.227445 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:33:26.227410    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-6cfcc85fbfgwqkr" event={"ID":"54f7c330-1562-4ba6-8cb5-9e509386ae04","Type":"ContainerDied","Data":"511090d28b5709858e97dbc51b7afdd01e68fe711f68f289f0594478d8985456"}
May 25 11:33:26.227551 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:33:26.227457    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-6cfcc85fbfgwqkr" event={"ID":"54f7c330-1562-4ba6-8cb5-9e509386ae04","Type":"ContainerDied","Data":"c6f64eb3bdd78283ad98b999b6fd73e8e3bf9f7c9bfca0890fecb20a451c95b8"}
May 25 11:33:26.227551 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:33:26.227480    2576 scope.go:117] "RemoveContainer" containerID="511090d28b5709858e97dbc51b7afdd01e68fe711f68f289f0594478d8985456"
May 25 11:33:26.236581 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:33:26.236563    2576 scope.go:117] "RemoveContainer" containerID="d2151d6147dee1fe766ac51b9eb8d63e439da83045377e84f994a7bd937f0032"
May 25 11:33:26.248190 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:33:26.248169    2576 scope.go:117] "RemoveContainer" containerID="511090d28b5709858e97dbc51b7afdd01e68fe711f68f289f0594478d8985456"
May 25 11:33:26.248493 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:33:26.248473    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"511090d28b5709858e97dbc51b7afdd01e68fe711f68f289f0594478d8985456\": container with ID starting with 511090d28b5709858e97dbc51b7afdd01e68fe711f68f289f0594478d8985456 not found: ID does not exist" containerID="511090d28b5709858e97dbc51b7afdd01e68fe711f68f289f0594478d8985456"
May 25 11:33:26.248546 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:33:26.248503    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"511090d28b5709858e97dbc51b7afdd01e68fe711f68f289f0594478d8985456"} err="failed to get container status \"511090d28b5709858e97dbc51b7afdd01e68fe711f68f289f0594478d8985456\": rpc error: code = NotFound desc = could not find container \"511090d28b5709858e97dbc51b7afdd01e68fe711f68f289f0594478d8985456\": container with ID starting with 511090d28b5709858e97dbc51b7afdd01e68fe711f68f289f0594478d8985456 not found: ID does not exist"
May 25 11:33:26.248546 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:33:26.248522    2576 scope.go:117] "RemoveContainer" containerID="d2151d6147dee1fe766ac51b9eb8d63e439da83045377e84f994a7bd937f0032"
May 25 11:33:26.248828 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:33:26.248793    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2151d6147dee1fe766ac51b9eb8d63e439da83045377e84f994a7bd937f0032\": container with ID starting with d2151d6147dee1fe766ac51b9eb8d63e439da83045377e84f994a7bd937f0032 not found: ID does not exist" containerID="d2151d6147dee1fe766ac51b9eb8d63e439da83045377e84f994a7bd937f0032"
May 25 11:33:26.248943 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:33:26.248827    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2151d6147dee1fe766ac51b9eb8d63e439da83045377e84f994a7bd937f0032"} err="failed to get container status \"d2151d6147dee1fe766ac51b9eb8d63e439da83045377e84f994a7bd937f0032\": rpc error: code = NotFound desc = could not find container \"d2151d6147dee1fe766ac51b9eb8d63e439da83045377e84f994a7bd937f0032\": container with ID starting with d2151d6147dee1fe766ac51b9eb8d63e439da83045377e84f994a7bd937f0032 not found: ID does not exist"
May 25 11:33:26.251029 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:33:26.251004    2576 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-6cfcc85fbfgwqkr"]
May 25 11:33:26.256471 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:33:26.256449    2576 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-6cfcc85fbfgwqkr"]
May 25 11:33:27.099481 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:33:27.099442    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="54f7c330-1562-4ba6-8cb5-9e509386ae04" path="/var/lib/kubelet/pods/54f7c330-1562-4ba6-8cb5-9e509386ae04/volumes"
May 25 11:33:41.279907 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:33:41.279866    2576 generic.go:358] "Generic (PLEG): container finished" podID="230c99ee-7273-42d7-be1d-a075fbbca797" containerID="f36bf81bbf28e340e41b2d0f21160e8fdbd5378ec277e8db4b49faf273189cc1" exitCode=0
May 25 11:33:41.280296 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:33:41.279919    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-6c579d5bdf-fh4bf" event={"ID":"230c99ee-7273-42d7-be1d-a075fbbca797","Type":"ContainerDied","Data":"f36bf81bbf28e340e41b2d0f21160e8fdbd5378ec277e8db4b49faf273189cc1"}
May 25 11:33:42.285052 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:33:42.285016    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-6c579d5bdf-fh4bf" event={"ID":"230c99ee-7273-42d7-be1d-a075fbbca797","Type":"ContainerStarted","Data":"a019a39dad6356fb747cc93ec4e7912fb0635819e59a00fadb4f34052b360f25"}
May 25 11:33:42.308633 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:33:42.308566    2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-6c579d5bdf-fh4bf" podStartSLOduration=37.308547924 podStartE2EDuration="37.308547924s" podCreationTimestamp="2026-05-25 11:33:05 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 11:33:42.307318665 +0000 UTC m=+967.825050127" watchObservedRunningTime="2026-05-25 11:33:42.308547924 +0000 UTC m=+967.826279381"
May 25 11:33:45.792241 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:33:45.792194    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-6c579d5bdf-fh4bf"
May 25 11:33:45.792241 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:33:45.792243    2576 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-6c579d5bdf-fh4bf"
May 25 11:33:45.793915 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:33:45.793875    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-6c579d5bdf-fh4bf" podUID="230c99ee-7273-42d7-be1d-a075fbbca797" containerName="main" probeResult="failure" output="Get \"https://10.133.0.32:8000/health\": dial tcp 10.133.0.32:8000: connect: connection refused"
May 25 11:33:55.791703 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:33:55.791655    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-6c579d5bdf-fh4bf" podUID="230c99ee-7273-42d7-be1d-a075fbbca797" containerName="main" probeResult="failure" output="Get \"https://10.133.0.32:8000/health\": dial tcp 10.133.0.32:8000: connect: connection refused"
May 25 11:34:05.791740 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:34:05.791694    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-6c579d5bdf-fh4bf" podUID="230c99ee-7273-42d7-be1d-a075fbbca797" containerName="main" probeResult="failure" output="Get \"https://10.133.0.32:8000/health\": dial tcp 10.133.0.32:8000: connect: connection refused"
May 25 11:34:15.791740 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:34:15.791694    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-6c579d5bdf-fh4bf" podUID="230c99ee-7273-42d7-be1d-a075fbbca797" containerName="main" probeResult="failure" output="Get \"https://10.133.0.32:8000/health\": dial tcp 10.133.0.32:8000: connect: connection refused"
May 25 11:34:25.791705 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:34:25.791660    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-6c579d5bdf-fh4bf" podUID="230c99ee-7273-42d7-be1d-a075fbbca797" containerName="main" probeResult="failure" output="Get \"https://10.133.0.32:8000/health\": dial tcp 10.133.0.32:8000: connect: connection refused"
May 25 11:34:35.792724 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:34:35.792674    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-6c579d5bdf-fh4bf" podUID="230c99ee-7273-42d7-be1d-a075fbbca797" containerName="main" probeResult="failure" output="Get \"https://10.133.0.32:8000/health\": dial tcp 10.133.0.32:8000: connect: connection refused"
May 25 11:34:45.792328 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:34:45.792265    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-6c579d5bdf-fh4bf" podUID="230c99ee-7273-42d7-be1d-a075fbbca797" containerName="main" probeResult="failure" output="Get \"https://10.133.0.32:8000/health\": dial tcp 10.133.0.32:8000: connect: connection refused"
May 25 11:34:55.791854 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:34:55.791813    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-6c579d5bdf-fh4bf" podUID="230c99ee-7273-42d7-be1d-a075fbbca797" containerName="main" probeResult="failure" output="Get \"https://10.133.0.32:8000/health\": dial tcp 10.133.0.32:8000: connect: connection refused"
May 25 11:35:05.792463 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:35:05.792416    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-6c579d5bdf-fh4bf" podUID="230c99ee-7273-42d7-be1d-a075fbbca797" containerName="main" probeResult="failure" output="Get \"https://10.133.0.32:8000/health\": dial tcp 10.133.0.32:8000: connect: connection refused"
May 25 11:35:15.801542 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:35:15.801508    2576 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-6c579d5bdf-fh4bf"
May 25 11:35:15.809073 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:35:15.809049    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-6c579d5bdf-fh4bf"
May 25 11:35:21.539769 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:35:21.539680    2576 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-6c579d5bdf-fh4bf"]
May 25 11:35:21.540246 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:35:21.539991    2576 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-6c579d5bdf-fh4bf" podUID="230c99ee-7273-42d7-be1d-a075fbbca797" containerName="main" containerID="cri-o://a019a39dad6356fb747cc93ec4e7912fb0635819e59a00fadb4f34052b360f25" gracePeriod=30
May 25 11:35:39.960255 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:35:39.960214    2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-cc6bc7969-2nj2j"]
May 25 11:35:39.960654 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:35:39.960575    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="54f7c330-1562-4ba6-8cb5-9e509386ae04" containerName="main"
May 25 11:35:39.960654 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:35:39.960586    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="54f7c330-1562-4ba6-8cb5-9e509386ae04" containerName="main"
May 25 11:35:39.960654 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:35:39.960609    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="54f7c330-1562-4ba6-8cb5-9e509386ae04" containerName="storage-initializer"
May 25 11:35:39.960654 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:35:39.960615    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="54f7c330-1562-4ba6-8cb5-9e509386ae04" containerName="storage-initializer"
May 25 11:35:39.960777 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:35:39.960664    2576 memory_manager.go:356] "RemoveStaleState removing state" podUID="54f7c330-1562-4ba6-8cb5-9e509386ae04" containerName="main"
May 25 11:35:39.962791 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:35:39.962771    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-cc6bc7969-2nj2j"
May 25 11:35:39.965707 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:35:39.965687    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"router-with-refs-test-kserve-self-signed-certs\""
May 25 11:35:39.977340 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:35:39.977308    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-cc6bc7969-2nj2j"]
May 25 11:35:40.104910 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:35:40.104878    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06d3b63c-4efc-4afd-8ae1-ed31f1559f71-model-cache\") pod \"router-with-refs-test-kserve-cc6bc7969-2nj2j\" (UID: \"06d3b63c-4efc-4afd-8ae1-ed31f1559f71\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-cc6bc7969-2nj2j"
May 25 11:35:40.105080 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:35:40.104929    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06d3b63c-4efc-4afd-8ae1-ed31f1559f71-tmp-dir\") pod \"router-with-refs-test-kserve-cc6bc7969-2nj2j\" (UID: \"06d3b63c-4efc-4afd-8ae1-ed31f1559f71\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-cc6bc7969-2nj2j"
May 25 11:35:40.105080 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:35:40.104991    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06d3b63c-4efc-4afd-8ae1-ed31f1559f71-tls-certs\") pod \"router-with-refs-test-kserve-cc6bc7969-2nj2j\" (UID: \"06d3b63c-4efc-4afd-8ae1-ed31f1559f71\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-cc6bc7969-2nj2j"
May 25 11:35:40.105080 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:35:40.105019    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbhjc\" (UniqueName: \"kubernetes.io/projected/06d3b63c-4efc-4afd-8ae1-ed31f1559f71-kube-api-access-sbhjc\") pod \"router-with-refs-test-kserve-cc6bc7969-2nj2j\" (UID: \"06d3b63c-4efc-4afd-8ae1-ed31f1559f71\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-cc6bc7969-2nj2j"
May 25 11:35:40.105189 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:35:40.105089    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/06d3b63c-4efc-4afd-8ae1-ed31f1559f71-kserve-provision-location\") pod \"router-with-refs-test-kserve-cc6bc7969-2nj2j\" (UID: \"06d3b63c-4efc-4afd-8ae1-ed31f1559f71\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-cc6bc7969-2nj2j"
May 25 11:35:40.105189 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:35:40.105127    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06d3b63c-4efc-4afd-8ae1-ed31f1559f71-dshm\") pod \"router-with-refs-test-kserve-cc6bc7969-2nj2j\" (UID: \"06d3b63c-4efc-4afd-8ae1-ed31f1559f71\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-cc6bc7969-2nj2j"
May 25 11:35:40.105189 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:35:40.105155    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06d3b63c-4efc-4afd-8ae1-ed31f1559f71-home\") pod \"router-with-refs-test-kserve-cc6bc7969-2nj2j\" (UID: \"06d3b63c-4efc-4afd-8ae1-ed31f1559f71\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-cc6bc7969-2nj2j"
May 25 11:35:40.206377 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:35:40.206336    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06d3b63c-4efc-4afd-8ae1-ed31f1559f71-tmp-dir\") pod \"router-with-refs-test-kserve-cc6bc7969-2nj2j\" (UID: \"06d3b63c-4efc-4afd-8ae1-ed31f1559f71\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-cc6bc7969-2nj2j"
May 25 11:35:40.206546 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:35:40.206482    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06d3b63c-4efc-4afd-8ae1-ed31f1559f71-tls-certs\") pod \"router-with-refs-test-kserve-cc6bc7969-2nj2j\" (UID: \"06d3b63c-4efc-4afd-8ae1-ed31f1559f71\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-cc6bc7969-2nj2j"
May 25 11:35:40.206546 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:35:40.206513    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-sbhjc\" (UniqueName: \"kubernetes.io/projected/06d3b63c-4efc-4afd-8ae1-ed31f1559f71-kube-api-access-sbhjc\") pod \"router-with-refs-test-kserve-cc6bc7969-2nj2j\" (UID: \"06d3b63c-4efc-4afd-8ae1-ed31f1559f71\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-cc6bc7969-2nj2j"
May 25 11:35:40.206546 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:35:40.206537    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/06d3b63c-4efc-4afd-8ae1-ed31f1559f71-kserve-provision-location\") pod \"router-with-refs-test-kserve-cc6bc7969-2nj2j\" (UID: \"06d3b63c-4efc-4afd-8ae1-ed31f1559f71\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-cc6bc7969-2nj2j"
May 25 11:35:40.206704 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:35:40.206559    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06d3b63c-4efc-4afd-8ae1-ed31f1559f71-dshm\") pod \"router-with-refs-test-kserve-cc6bc7969-2nj2j\" (UID: \"06d3b63c-4efc-4afd-8ae1-ed31f1559f71\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-cc6bc7969-2nj2j"
May 25 11:35:40.206704 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:35:40.206589    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06d3b63c-4efc-4afd-8ae1-ed31f1559f71-home\") pod \"router-with-refs-test-kserve-cc6bc7969-2nj2j\" (UID: \"06d3b63c-4efc-4afd-8ae1-ed31f1559f71\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-cc6bc7969-2nj2j"
May 25 11:35:40.206704 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:35:40.206635    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06d3b63c-4efc-4afd-8ae1-ed31f1559f71-model-cache\") pod \"router-with-refs-test-kserve-cc6bc7969-2nj2j\" (UID: \"06d3b63c-4efc-4afd-8ae1-ed31f1559f71\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-cc6bc7969-2nj2j"
May 25 11:35:40.206872 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:35:40.206842    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06d3b63c-4efc-4afd-8ae1-ed31f1559f71-tmp-dir\") pod \"router-with-refs-test-kserve-cc6bc7969-2nj2j\" (UID: \"06d3b63c-4efc-4afd-8ae1-ed31f1559f71\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-cc6bc7969-2nj2j"
May 25 11:35:40.206958 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:35:40.206936    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/06d3b63c-4efc-4afd-8ae1-ed31f1559f71-kserve-provision-location\") pod \"router-with-refs-test-kserve-cc6bc7969-2nj2j\" (UID: \"06d3b63c-4efc-4afd-8ae1-ed31f1559f71\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-cc6bc7969-2nj2j"
May 25 11:35:40.207013 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:35:40.206962    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06d3b63c-4efc-4afd-8ae1-ed31f1559f71-home\") pod \"router-with-refs-test-kserve-cc6bc7969-2nj2j\" (UID: \"06d3b63c-4efc-4afd-8ae1-ed31f1559f71\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-cc6bc7969-2nj2j"
May 25 11:35:40.207093 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:35:40.207074    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06d3b63c-4efc-4afd-8ae1-ed31f1559f71-model-cache\") pod \"router-with-refs-test-kserve-cc6bc7969-2nj2j\" (UID: \"06d3b63c-4efc-4afd-8ae1-ed31f1559f71\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-cc6bc7969-2nj2j"
May 25 11:35:40.208826 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:35:40.208803    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06d3b63c-4efc-4afd-8ae1-ed31f1559f71-dshm\") pod \"router-with-refs-test-kserve-cc6bc7969-2nj2j\" (UID: \"06d3b63c-4efc-4afd-8ae1-ed31f1559f71\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-cc6bc7969-2nj2j"
May 25 11:35:40.208985 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:35:40.208966    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06d3b63c-4efc-4afd-8ae1-ed31f1559f71-tls-certs\") pod \"router-with-refs-test-kserve-cc6bc7969-2nj2j\" (UID: \"06d3b63c-4efc-4afd-8ae1-ed31f1559f71\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-cc6bc7969-2nj2j"
May 25 11:35:40.220113 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:35:40.220056    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbhjc\" (UniqueName: \"kubernetes.io/projected/06d3b63c-4efc-4afd-8ae1-ed31f1559f71-kube-api-access-sbhjc\") pod \"router-with-refs-test-kserve-cc6bc7969-2nj2j\" (UID: \"06d3b63c-4efc-4afd-8ae1-ed31f1559f71\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-cc6bc7969-2nj2j"
May 25 11:35:40.273211 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:35:40.273178    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-cc6bc7969-2nj2j"
May 25 11:35:40.449171 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:35:40.449147    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-cc6bc7969-2nj2j"]
May 25 11:35:40.451639 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:35:40.451611    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06d3b63c_4efc_4afd_8ae1_ed31f1559f71.slice/crio-bb218ced20e29b616a018e6bbeff077483e9ee17909760416e2d0765148540eb WatchSource:0}: Error finding container bb218ced20e29b616a018e6bbeff077483e9ee17909760416e2d0765148540eb: Status 404 returned error can't find the container with id bb218ced20e29b616a018e6bbeff077483e9ee17909760416e2d0765148540eb
May 25 11:35:40.453470 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:35:40.453452    2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 25 11:35:40.696592 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:35:40.696547    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-cc6bc7969-2nj2j" event={"ID":"06d3b63c-4efc-4afd-8ae1-ed31f1559f71","Type":"ContainerStarted","Data":"a4fa78b1c647f824d94d03a10f9aee72ea26c27b29c305d928361ae101039c9f"}
May 25 11:35:40.696592 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:35:40.696596    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-cc6bc7969-2nj2j" event={"ID":"06d3b63c-4efc-4afd-8ae1-ed31f1559f71","Type":"ContainerStarted","Data":"bb218ced20e29b616a018e6bbeff077483e9ee17909760416e2d0765148540eb"}
May 25 11:35:51.737139 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:35:51.736048    2576 generic.go:358] "Generic (PLEG): container finished" podID="06d3b63c-4efc-4afd-8ae1-ed31f1559f71" containerID="a4fa78b1c647f824d94d03a10f9aee72ea26c27b29c305d928361ae101039c9f" exitCode=0
May 25 11:35:51.737139 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:35:51.736154    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-cc6bc7969-2nj2j" event={"ID":"06d3b63c-4efc-4afd-8ae1-ed31f1559f71","Type":"ContainerDied","Data":"a4fa78b1c647f824d94d03a10f9aee72ea26c27b29c305d928361ae101039c9f"}
May 25 11:35:51.740912 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:35:51.740887    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_custom-route-timeout-test-kserve-6c579d5bdf-fh4bf_230c99ee-7273-42d7-be1d-a075fbbca797/main/0.log"
May 25 11:35:51.741336 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:35:51.741314    2576 generic.go:358] "Generic (PLEG): container finished" podID="230c99ee-7273-42d7-be1d-a075fbbca797" containerID="a019a39dad6356fb747cc93ec4e7912fb0635819e59a00fadb4f34052b360f25" exitCode=137
May 25 11:35:51.741411 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:35:51.741355    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-6c579d5bdf-fh4bf" event={"ID":"230c99ee-7273-42d7-be1d-a075fbbca797","Type":"ContainerDied","Data":"a019a39dad6356fb747cc93ec4e7912fb0635819e59a00fadb4f34052b360f25"}
May 25 11:35:51.790107 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:35:51.789858    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_custom-route-timeout-test-kserve-6c579d5bdf-fh4bf_230c99ee-7273-42d7-be1d-a075fbbca797/main/0.log"
May 25 11:35:51.790465 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:35:51.790445    2576 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-6c579d5bdf-fh4bf"
May 25 11:35:51.907323 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:35:51.907263    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/230c99ee-7273-42d7-be1d-a075fbbca797-dshm\") pod \"230c99ee-7273-42d7-be1d-a075fbbca797\" (UID: \"230c99ee-7273-42d7-be1d-a075fbbca797\") "
May 25 11:35:51.907509 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:35:51.907356    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hjrx4\" (UniqueName: \"kubernetes.io/projected/230c99ee-7273-42d7-be1d-a075fbbca797-kube-api-access-hjrx4\") pod \"230c99ee-7273-42d7-be1d-a075fbbca797\" (UID: \"230c99ee-7273-42d7-be1d-a075fbbca797\") "
May 25 11:35:51.907509 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:35:51.907413    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/230c99ee-7273-42d7-be1d-a075fbbca797-home\") pod \"230c99ee-7273-42d7-be1d-a075fbbca797\" (UID: \"230c99ee-7273-42d7-be1d-a075fbbca797\") "
May 25 11:35:51.907509 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:35:51.907439    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/230c99ee-7273-42d7-be1d-a075fbbca797-tls-certs\") pod \"230c99ee-7273-42d7-be1d-a075fbbca797\" (UID: \"230c99ee-7273-42d7-be1d-a075fbbca797\") "
May 25 11:35:51.907509 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:35:51.907494    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/230c99ee-7273-42d7-be1d-a075fbbca797-tmp-dir\") pod \"230c99ee-7273-42d7-be1d-a075fbbca797\" (UID: \"230c99ee-7273-42d7-be1d-a075fbbca797\") "
May 25 11:35:51.907726 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:35:51.907515    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/230c99ee-7273-42d7-be1d-a075fbbca797-model-cache\") pod \"230c99ee-7273-42d7-be1d-a075fbbca797\" (UID: \"230c99ee-7273-42d7-be1d-a075fbbca797\") "
May 25 11:35:51.907726 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:35:51.907553    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/230c99ee-7273-42d7-be1d-a075fbbca797-kserve-provision-location\") pod \"230c99ee-7273-42d7-be1d-a075fbbca797\" (UID: \"230c99ee-7273-42d7-be1d-a075fbbca797\") "
May 25 11:35:51.908069 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:35:51.907957    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/230c99ee-7273-42d7-be1d-a075fbbca797-model-cache" (OuterVolumeSpecName: "model-cache") pod "230c99ee-7273-42d7-be1d-a075fbbca797" (UID: "230c99ee-7273-42d7-be1d-a075fbbca797"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:35:51.908320 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:35:51.908262    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/230c99ee-7273-42d7-be1d-a075fbbca797-home" (OuterVolumeSpecName: "home") pod "230c99ee-7273-42d7-be1d-a075fbbca797" (UID: "230c99ee-7273-42d7-be1d-a075fbbca797"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:35:51.909906 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:35:51.909878    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/230c99ee-7273-42d7-be1d-a075fbbca797-dshm" (OuterVolumeSpecName: "dshm") pod "230c99ee-7273-42d7-be1d-a075fbbca797" (UID: "230c99ee-7273-42d7-be1d-a075fbbca797"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:35:51.910022 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:35:51.909878    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/230c99ee-7273-42d7-be1d-a075fbbca797-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "230c99ee-7273-42d7-be1d-a075fbbca797" (UID: "230c99ee-7273-42d7-be1d-a075fbbca797"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 11:35:51.910088 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:35:51.910022    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/230c99ee-7273-42d7-be1d-a075fbbca797-kube-api-access-hjrx4" (OuterVolumeSpecName: "kube-api-access-hjrx4") pod "230c99ee-7273-42d7-be1d-a075fbbca797" (UID: "230c99ee-7273-42d7-be1d-a075fbbca797"). InnerVolumeSpecName "kube-api-access-hjrx4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 11:35:51.919675 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:35:51.919620    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/230c99ee-7273-42d7-be1d-a075fbbca797-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "230c99ee-7273-42d7-be1d-a075fbbca797" (UID: "230c99ee-7273-42d7-be1d-a075fbbca797"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:35:51.970288 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:35:51.970228    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/230c99ee-7273-42d7-be1d-a075fbbca797-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "230c99ee-7273-42d7-be1d-a075fbbca797" (UID: "230c99ee-7273-42d7-be1d-a075fbbca797"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:35:52.009124 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:35:52.009090    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/230c99ee-7273-42d7-be1d-a075fbbca797-tmp-dir\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
May 25 11:35:52.009124 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:35:52.009120    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/230c99ee-7273-42d7-be1d-a075fbbca797-model-cache\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
May 25 11:35:52.009124 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:35:52.009132    2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/230c99ee-7273-42d7-be1d-a075fbbca797-kserve-provision-location\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
May 25 11:35:52.009377 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:35:52.009141    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/230c99ee-7273-42d7-be1d-a075fbbca797-dshm\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
May 25 11:35:52.009377 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:35:52.009152    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hjrx4\" (UniqueName: \"kubernetes.io/projected/230c99ee-7273-42d7-be1d-a075fbbca797-kube-api-access-hjrx4\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
May 25 11:35:52.009377 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:35:52.009160    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/230c99ee-7273-42d7-be1d-a075fbbca797-home\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
May 25 11:35:52.009377 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:35:52.009169    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/230c99ee-7273-42d7-be1d-a075fbbca797-tls-certs\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
May 25 11:35:52.747581 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:35:52.747540    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-cc6bc7969-2nj2j" event={"ID":"06d3b63c-4efc-4afd-8ae1-ed31f1559f71","Type":"ContainerStarted","Data":"2b664f44b1ea8e48f207588fd81da6f1c85bddda2a4a4641ae3f9a921467f193"}
May 25 11:35:52.748857 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:35:52.748833    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_custom-route-timeout-test-kserve-6c579d5bdf-fh4bf_230c99ee-7273-42d7-be1d-a075fbbca797/main/0.log"
May 25 11:35:52.749146 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:35:52.749125    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-6c579d5bdf-fh4bf" event={"ID":"230c99ee-7273-42d7-be1d-a075fbbca797","Type":"ContainerDied","Data":"23d72c1adf82d036c21849a7cf084c0a4d838246d06e7e7b40af489aa02ae9b6"}
May 25 11:35:52.749232 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:35:52.749162    2576 scope.go:117] "RemoveContainer" containerID="a019a39dad6356fb747cc93ec4e7912fb0635819e59a00fadb4f34052b360f25"
May 25 11:35:52.749232 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:35:52.749177    2576 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-6c579d5bdf-fh4bf"
May 25 11:35:52.758159 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:35:52.758143    2576 scope.go:117] "RemoveContainer" containerID="f36bf81bbf28e340e41b2d0f21160e8fdbd5378ec277e8db4b49faf273189cc1"
May 25 11:35:52.775810 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:35:52.775742    2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-cc6bc7969-2nj2j" podStartSLOduration=13.775723764 podStartE2EDuration="13.775723764s" podCreationTimestamp="2026-05-25 11:35:39 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 11:35:52.774974416 +0000 UTC m=+1098.292705872" watchObservedRunningTime="2026-05-25 11:35:52.775723764 +0000 UTC m=+1098.293455221"
May 25 11:35:52.792898 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:35:52.792870    2576 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-6c579d5bdf-fh4bf"]
May 25 11:35:52.802259 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:35:52.802229    2576 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-6c579d5bdf-fh4bf"]
May 25 11:35:53.099906 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:35:53.099821    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="230c99ee-7273-42d7-be1d-a075fbbca797" path="/var/lib/kubelet/pods/230c99ee-7273-42d7-be1d-a075fbbca797/volumes"
May 25 11:36:00.273723 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:36:00.273684    2576 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-cc6bc7969-2nj2j"
May 25 11:36:00.273723 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:36:00.273732    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-cc6bc7969-2nj2j"
May 25 11:36:00.275436 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:36:00.275409    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-cc6bc7969-2nj2j" podUID="06d3b63c-4efc-4afd-8ae1-ed31f1559f71" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/health\": dial tcp 10.133.0.33:8000: connect: connection refused"
May 25 11:36:10.274567 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:36:10.274517    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-cc6bc7969-2nj2j" podUID="06d3b63c-4efc-4afd-8ae1-ed31f1559f71" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/health\": dial tcp 10.133.0.33:8000: connect: connection refused"
May 25 11:36:20.274228 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:36:20.274185    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-cc6bc7969-2nj2j" podUID="06d3b63c-4efc-4afd-8ae1-ed31f1559f71" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/health\": dial tcp 10.133.0.33:8000: connect: connection refused"
May 25 11:36:30.273956 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:36:30.273906    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-cc6bc7969-2nj2j" podUID="06d3b63c-4efc-4afd-8ae1-ed31f1559f71" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/health\": dial tcp 10.133.0.33:8000: connect: connection refused"
May 25 11:36:40.273740 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:36:40.273696    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-cc6bc7969-2nj2j" podUID="06d3b63c-4efc-4afd-8ae1-ed31f1559f71" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/health\": dial tcp 10.133.0.33:8000: connect: connection refused"
May 25 11:36:50.274217 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:36:50.274174    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-cc6bc7969-2nj2j" podUID="06d3b63c-4efc-4afd-8ae1-ed31f1559f71" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/health\": dial tcp 10.133.0.33:8000: connect: connection refused"
May 25 11:37:00.274446 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:37:00.274403    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-cc6bc7969-2nj2j" podUID="06d3b63c-4efc-4afd-8ae1-ed31f1559f71" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/health\": dial tcp 10.133.0.33:8000: connect: connection refused"
May 25 11:37:10.274080 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:37:10.274009    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-cc6bc7969-2nj2j" podUID="06d3b63c-4efc-4afd-8ae1-ed31f1559f71" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/health\": dial tcp 10.133.0.33:8000: connect: connection refused"
May 25 11:37:20.274091 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:37:20.274044    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-cc6bc7969-2nj2j" podUID="06d3b63c-4efc-4afd-8ae1-ed31f1559f71" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/health\": dial tcp 10.133.0.33:8000: connect: connection refused"
May 25 11:37:30.284358 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:37:30.284327    2576 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-cc6bc7969-2nj2j"
May 25 11:37:30.292488 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:37:30.292463    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-cc6bc7969-2nj2j"
May 25 11:37:35.091346 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:37:35.091313    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h6b5t_12b45353-8444-41fc-a8ed-fb691764eb63/ovn-acl-logging/0.log"
May 25 11:37:35.091915 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:37:35.091894    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h6b5t_12b45353-8444-41fc-a8ed-fb691764eb63/ovn-acl-logging/0.log"
May 25 11:37:36.413600 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:37:36.413562    2576 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-cc6bc7969-2nj2j"]
May 25 11:37:36.414001 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:37:36.413903    2576 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-cc6bc7969-2nj2j" podUID="06d3b63c-4efc-4afd-8ae1-ed31f1559f71" containerName="main" containerID="cri-o://2b664f44b1ea8e48f207588fd81da6f1c85bddda2a4a4641ae3f9a921467f193" gracePeriod=30
May 25 11:37:40.738973 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:37:40.738922    2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-66b9664c88slmlh"]
May 25 11:37:40.739362 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:37:40.739311    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="230c99ee-7273-42d7-be1d-a075fbbca797" containerName="storage-initializer"
May 25 11:37:40.739362 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:37:40.739327    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="230c99ee-7273-42d7-be1d-a075fbbca797" containerName="storage-initializer"
May 25 11:37:40.739362 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:37:40.739352    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="230c99ee-7273-42d7-be1d-a075fbbca797" containerName="main"
May 25 11:37:40.739362 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:37:40.739361    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="230c99ee-7273-42d7-be1d-a075fbbca797" containerName="main"
May 25 11:37:40.739669 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:37:40.739463    2576 memory_manager.go:356] "RemoveStaleState removing state" podUID="230c99ee-7273-42d7-be1d-a075fbbca797" containerName="main"
May 25 11:37:40.743168 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:37:40.743150    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-66b9664c88slmlh"
May 25 11:37:40.745777 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:37:40.745751    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv8f1a6f044e8c7a4d31a250e0c4861caf-kserve-self-signed-certs\""
May 25 11:37:40.745887 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:37:40.745761    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-dockercfg-ksdwf\""
May 25 11:37:40.746788 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:37:40.746762    2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-cdng2p5"]
May 25 11:37:40.750720 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:37:40.750699    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-cdng2p5"
May 25 11:37:40.756509 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:37:40.756483    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-66b9664c88slmlh"]
May 25 11:37:40.765842 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:37:40.765819    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-cdng2p5"]
May 25 11:37:40.802754 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:37:40.802721    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a33d5fe5-3633-466f-888d-5250a697bed0-dshm\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-66b9664c88slmlh\" (UID: \"a33d5fe5-3633-466f-888d-5250a697bed0\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-66b9664c88slmlh"
May 25 11:37:40.802882 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:37:40.802841    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a33d5fe5-3633-466f-888d-5250a697bed0-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-66b9664c88slmlh\" (UID: \"a33d5fe5-3633-466f-888d-5250a697bed0\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-66b9664c88slmlh"
May 25 11:37:40.802951 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:37:40.802897    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a33d5fe5-3633-466f-888d-5250a697bed0-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-66b9664c88slmlh\" (UID: \"a33d5fe5-3633-466f-888d-5250a697bed0\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-66b9664c88slmlh"
May 25 11:37:40.802951 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:37:40.802934    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a33d5fe5-3633-466f-888d-5250a697bed0-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-66b9664c88slmlh\" (UID: \"a33d5fe5-3633-466f-888d-5250a697bed0\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-66b9664c88slmlh"
May 25 11:37:40.803048 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:37:40.803003    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a33d5fe5-3633-466f-888d-5250a697bed0-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-66b9664c88slmlh\" (UID: \"a33d5fe5-3633-466f-888d-5250a697bed0\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-66b9664c88slmlh"
May 25 11:37:40.803091 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:37:40.803046    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24gzx\" (UniqueName: \"kubernetes.io/projected/a33d5fe5-3633-466f-888d-5250a697bed0-kube-api-access-24gzx\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-66b9664c88slmlh\" (UID: \"a33d5fe5-3633-466f-888d-5250a697bed0\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-66b9664c88slmlh"
May 25 11:37:40.803126 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:37:40.803099    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a33d5fe5-3633-466f-888d-5250a697bed0-home\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-66b9664c88slmlh\" (UID: \"a33d5fe5-3633-466f-888d-5250a697bed0\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-66b9664c88slmlh"
May 25 11:37:40.904242 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:37:40.904215    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/599e3fe7-d07d-405b-820a-42e866f1636d-home\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-cdng2p5\" (UID: \"599e3fe7-d07d-405b-820a-42e866f1636d\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-cdng2p5"
May 25 11:37:40.904377 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:37:40.904245    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/599e3fe7-d07d-405b-820a-42e866f1636d-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-cdng2p5\" (UID: \"599e3fe7-d07d-405b-820a-42e866f1636d\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-cdng2p5"
May 25 11:37:40.904377 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:37:40.904267    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a33d5fe5-3633-466f-888d-5250a697bed0-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-66b9664c88slmlh\" (UID: \"a33d5fe5-3633-466f-888d-5250a697bed0\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-66b9664c88slmlh"
May 25 11:37:40.904377 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:37:40.904355    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-24gzx\" (UniqueName: \"kubernetes.io/projected/a33d5fe5-3633-466f-888d-5250a697bed0-kube-api-access-24gzx\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-66b9664c88slmlh\" (UID: \"a33d5fe5-3633-466f-888d-5250a697bed0\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-66b9664c88slmlh"
May 25 11:37:40.904509 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:37:40.904385    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/599e3fe7-d07d-405b-820a-42e866f1636d-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-cdng2p5\" (UID: \"599e3fe7-d07d-405b-820a-42e866f1636d\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-cdng2p5"
May 25 11:37:40.904509 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:37:40.904401    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/599e3fe7-d07d-405b-820a-42e866f1636d-dshm\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-cdng2p5\" (UID: \"599e3fe7-d07d-405b-820a-42e866f1636d\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-cdng2p5"
May 25 11:37:40.904509 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:37:40.904425    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/599e3fe7-d07d-405b-820a-42e866f1636d-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-cdng2p5\" (UID: \"599e3fe7-d07d-405b-820a-42e866f1636d\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-cdng2p5"
May 25 11:37:40.904509 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:37:40.904454    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26qwq\" (UniqueName: \"kubernetes.io/projected/599e3fe7-d07d-405b-820a-42e866f1636d-kube-api-access-26qwq\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-cdng2p5\" (UID: \"599e3fe7-d07d-405b-820a-42e866f1636d\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-cdng2p5"
May 25 11:37:40.904509 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:37:40.904484    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a33d5fe5-3633-466f-888d-5250a697bed0-home\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-66b9664c88slmlh\" (UID: \"a33d5fe5-3633-466f-888d-5250a697bed0\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-66b9664c88slmlh"
May 25 11:37:40.904703 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:37:40.904541    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a33d5fe5-3633-466f-888d-5250a697bed0-dshm\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-66b9664c88slmlh\" (UID: \"a33d5fe5-3633-466f-888d-5250a697bed0\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-66b9664c88slmlh"
May 25 11:37:40.904703 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:37:40.904563    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/599e3fe7-d07d-405b-820a-42e866f1636d-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-cdng2p5\" (UID: \"599e3fe7-d07d-405b-820a-42e866f1636d\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-cdng2p5"
May 25 11:37:40.904703 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:37:40.904636    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a33d5fe5-3633-466f-888d-5250a697bed0-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-66b9664c88slmlh\" (UID: \"a33d5fe5-3633-466f-888d-5250a697bed0\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-66b9664c88slmlh"
May 25 11:37:40.904703 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:37:40.904680    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a33d5fe5-3633-466f-888d-5250a697bed0-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-66b9664c88slmlh\" (UID: \"a33d5fe5-3633-466f-888d-5250a697bed0\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-66b9664c88slmlh"
May 25 11:37:40.904703 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:37:40.904689    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a33d5fe5-3633-466f-888d-5250a697bed0-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-66b9664c88slmlh\" (UID: \"a33d5fe5-3633-466f-888d-5250a697bed0\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-66b9664c88slmlh"
May 25 11:37:40.904971 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:37:40.904726    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a33d5fe5-3633-466f-888d-5250a697bed0-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-66b9664c88slmlh\" (UID: \"a33d5fe5-3633-466f-888d-5250a697bed0\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-66b9664c88slmlh"
May 25 11:37:40.904971 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:37:40.904737    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a33d5fe5-3633-466f-888d-5250a697bed0-home\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-66b9664c88slmlh\" (UID: \"a33d5fe5-3633-466f-888d-5250a697bed0\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-66b9664c88slmlh"
May 25 11:37:40.904971 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:37:40.904944    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a33d5fe5-3633-466f-888d-5250a697bed0-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-66b9664c88slmlh\" (UID: \"a33d5fe5-3633-466f-888d-5250a697bed0\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-66b9664c88slmlh"
May 25 11:37:40.904971 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:37:40.904952    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a33d5fe5-3633-466f-888d-5250a697bed0-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-66b9664c88slmlh\" (UID: \"a33d5fe5-3633-466f-888d-5250a697bed0\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-66b9664c88slmlh"
May 25 11:37:40.906598 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:37:40.906581    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a33d5fe5-3633-466f-888d-5250a697bed0-dshm\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-66b9664c88slmlh\" (UID: \"a33d5fe5-3633-466f-888d-5250a697bed0\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-66b9664c88slmlh"
May 25 11:37:40.906922 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:37:40.906906    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a33d5fe5-3633-466f-888d-5250a697bed0-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-66b9664c88slmlh\" (UID: \"a33d5fe5-3633-466f-888d-5250a697bed0\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-66b9664c88slmlh"
May 25 11:37:40.913368 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:37:40.913339    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-24gzx\" (UniqueName: \"kubernetes.io/projected/a33d5fe5-3633-466f-888d-5250a697bed0-kube-api-access-24gzx\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-66b9664c88slmlh\" (UID: \"a33d5fe5-3633-466f-888d-5250a697bed0\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-66b9664c88slmlh"
May 25 11:37:41.005676 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:37:41.005607    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/599e3fe7-d07d-405b-820a-42e866f1636d-home\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-cdng2p5\" (UID: \"599e3fe7-d07d-405b-820a-42e866f1636d\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-cdng2p5"
May 25 11:37:41.005676 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:37:41.005640    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/599e3fe7-d07d-405b-820a-42e866f1636d-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-cdng2p5\" (UID: \"599e3fe7-d07d-405b-820a-42e866f1636d\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-cdng2p5"
May 25 11:37:41.005676 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:37:41.005662    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/599e3fe7-d07d-405b-820a-42e866f1636d-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-cdng2p5\" (UID: \"599e3fe7-d07d-405b-820a-42e866f1636d\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-cdng2p5"
May 25 11:37:41.005912 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:37:41.005682    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/599e3fe7-d07d-405b-820a-42e866f1636d-dshm\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-cdng2p5\" (UID: \"599e3fe7-d07d-405b-820a-42e866f1636d\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-cdng2p5"
May 25 11:37:41.005912 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:37:41.005709    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/599e3fe7-d07d-405b-820a-42e866f1636d-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-cdng2p5\" (UID: \"599e3fe7-d07d-405b-820a-42e866f1636d\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-cdng2p5"
May 25 11:37:41.005912 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:37:41.005744    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-26qwq\" (UniqueName: \"kubernetes.io/projected/599e3fe7-d07d-405b-820a-42e866f1636d-kube-api-access-26qwq\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-cdng2p5\" (UID: \"599e3fe7-d07d-405b-820a-42e866f1636d\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-cdng2p5"
May 25 11:37:41.005912 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:37:41.005822    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/599e3fe7-d07d-405b-820a-42e866f1636d-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-cdng2p5\" (UID: \"599e3fe7-d07d-405b-820a-42e866f1636d\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-cdng2p5"
May 25 11:37:41.006126 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:37:41.006047    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/599e3fe7-d07d-405b-820a-42e866f1636d-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-cdng2p5\" (UID: \"599e3fe7-d07d-405b-820a-42e866f1636d\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-cdng2p5"
May 25 11:37:41.006126 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:37:41.006054    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/599e3fe7-d07d-405b-820a-42e866f1636d-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-cdng2p5\" (UID: \"599e3fe7-d07d-405b-820a-42e866f1636d\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-cdng2p5"
May 25 11:37:41.006317 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:37:41.006294    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/599e3fe7-d07d-405b-820a-42e866f1636d-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-cdng2p5\" (UID: \"599e3fe7-d07d-405b-820a-42e866f1636d\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-cdng2p5"
May 25 11:37:41.006391 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:37:41.006368    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/599e3fe7-d07d-405b-820a-42e866f1636d-home\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-cdng2p5\" (UID: \"599e3fe7-d07d-405b-820a-42e866f1636d\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-cdng2p5"
May 25 11:37:41.007862 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:37:41.007845    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/599e3fe7-d07d-405b-820a-42e866f1636d-dshm\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-cdng2p5\" (UID: \"599e3fe7-d07d-405b-820a-42e866f1636d\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-cdng2p5"
May 25 11:37:41.008267 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:37:41.008247    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/599e3fe7-d07d-405b-820a-42e866f1636d-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-cdng2p5\" (UID: \"599e3fe7-d07d-405b-820a-42e866f1636d\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-cdng2p5"
May 25 11:37:41.023893 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:37:41.023870    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-26qwq\" (UniqueName: \"kubernetes.io/projected/599e3fe7-d07d-405b-820a-42e866f1636d-kube-api-access-26qwq\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-cdng2p5\" (UID: \"599e3fe7-d07d-405b-820a-42e866f1636d\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-cdng2p5"
May 25 11:37:41.053918 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:37:41.053895    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-66b9664c88slmlh"
May 25 11:37:41.062662 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:37:41.062641    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-cdng2p5"
May 25 11:37:41.202891 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:37:41.202868    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-66b9664c88slmlh"]
May 25 11:37:41.205430 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:37:41.205404    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda33d5fe5_3633_466f_888d_5250a697bed0.slice/crio-861282ee2ecf155c4ad5bfa8bcd1437cc482d5197582fead2cb5b70709118ce0 WatchSource:0}: Error finding container 861282ee2ecf155c4ad5bfa8bcd1437cc482d5197582fead2cb5b70709118ce0: Status 404 returned error can't find the container with id 861282ee2ecf155c4ad5bfa8bcd1437cc482d5197582fead2cb5b70709118ce0
May 25 11:37:41.218576 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:37:41.218559    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-cdng2p5"]
May 25 11:37:41.221065 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:37:41.221034    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod599e3fe7_d07d_405b_820a_42e866f1636d.slice/crio-2e2c0301a72e6ee32ed662a2348c27e88da92c2469df8bb43d6c73a3a74eb81a WatchSource:0}: Error finding container 2e2c0301a72e6ee32ed662a2348c27e88da92c2469df8bb43d6c73a3a74eb81a: Status 404 returned error can't find the container with id 2e2c0301a72e6ee32ed662a2348c27e88da92c2469df8bb43d6c73a3a74eb81a
May 25 11:37:42.141732 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:37:42.141698    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-cdng2p5" event={"ID":"599e3fe7-d07d-405b-820a-42e866f1636d","Type":"ContainerStarted","Data":"cd56801d6efc44f41a3e07389b797c791b08761183bcb479bd6b383662dd435a"}
May 25 11:37:42.142107 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:37:42.141742    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-cdng2p5" event={"ID":"599e3fe7-d07d-405b-820a-42e866f1636d","Type":"ContainerStarted","Data":"2e2c0301a72e6ee32ed662a2348c27e88da92c2469df8bb43d6c73a3a74eb81a"}
May 25 11:37:42.143133 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:37:42.143074    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-66b9664c88slmlh" event={"ID":"a33d5fe5-3633-466f-888d-5250a697bed0","Type":"ContainerStarted","Data":"ec8382e356e0a5f26fb5ad1ede5104af1e35c44b9c7116f1bf0b54f7079f7b23"}
May 25 11:37:42.143133 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:37:42.143108    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-66b9664c88slmlh" event={"ID":"a33d5fe5-3633-466f-888d-5250a697bed0","Type":"ContainerStarted","Data":"861282ee2ecf155c4ad5bfa8bcd1437cc482d5197582fead2cb5b70709118ce0"}
May 25 11:37:42.143343 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:37:42.143313    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-66b9664c88slmlh"
May 25 11:37:43.147781 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:37:43.147746    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-66b9664c88slmlh" event={"ID":"a33d5fe5-3633-466f-888d-5250a697bed0","Type":"ContainerStarted","Data":"01220baf28b056dfe7bc5f90266fc51d887398914ac3ccf6dcce2e09c250053f"}
May 25 11:37:46.159260 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:37:46.159223    2576 generic.go:358] "Generic (PLEG): container finished" podID="599e3fe7-d07d-405b-820a-42e866f1636d" containerID="cd56801d6efc44f41a3e07389b797c791b08761183bcb479bd6b383662dd435a" exitCode=0
May 25 11:37:46.159640 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:37:46.159272    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-cdng2p5" event={"ID":"599e3fe7-d07d-405b-820a-42e866f1636d","Type":"ContainerDied","Data":"cd56801d6efc44f41a3e07389b797c791b08761183bcb479bd6b383662dd435a"}
May 25 11:37:47.164594 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:37:47.164553    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-cdng2p5" event={"ID":"599e3fe7-d07d-405b-820a-42e866f1636d","Type":"ContainerStarted","Data":"7e4b640b0cf7db34fa537cb82fbc2f323e65ed91006c262cc7e0b84a2148d0c1"}
May 25 11:37:47.166272 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:37:47.166242    2576 generic.go:358] "Generic (PLEG): container finished" podID="a33d5fe5-3633-466f-888d-5250a697bed0" containerID="01220baf28b056dfe7bc5f90266fc51d887398914ac3ccf6dcce2e09c250053f" exitCode=0
May 25 11:37:47.166435 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:37:47.166320    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-66b9664c88slmlh" event={"ID":"a33d5fe5-3633-466f-888d-5250a697bed0","Type":"ContainerDied","Data":"01220baf28b056dfe7bc5f90266fc51d887398914ac3ccf6dcce2e09c250053f"}
May 25 11:37:47.195029 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:37:47.194981    2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-cdng2p5" podStartSLOduration=7.194966585 podStartE2EDuration="7.194966585s" podCreationTimestamp="2026-05-25 11:37:40 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 11:37:47.193965553 +0000 UTC m=+1212.711697009" watchObservedRunningTime="2026-05-25 11:37:47.194966585 +0000 UTC m=+1212.712698040"
May 25 11:37:48.180602 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:37:48.180563    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-66b9664c88slmlh" event={"ID":"a33d5fe5-3633-466f-888d-5250a697bed0","Type":"ContainerStarted","Data":"d2dfed07f97d309cb9295959a696a162ecbf0c37c84deb20a02c172733ed5fc5"}
May 25 11:37:48.205019 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:37:48.204954    2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-66b9664c88slmlh" podStartSLOduration=7.346414273 podStartE2EDuration="8.204935356s" podCreationTimestamp="2026-05-25 11:37:40 +0000 UTC" firstStartedPulling="2026-05-25 11:37:41.207480991 +0000 UTC m=+1206.725212425" lastFinishedPulling="2026-05-25 11:37:42.066002065 +0000 UTC m=+1207.583733508" observedRunningTime="2026-05-25 11:37:48.202704919 +0000 UTC m=+1213.720436386" watchObservedRunningTime="2026-05-25 11:37:48.204935356 +0000 UTC m=+1213.722666812"
May 25 11:37:51.054977 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:37:51.054928    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-66b9664c88slmlh"
May 25 11:37:51.055411 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:37:51.054990    2576 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-66b9664c88slmlh"
May 25 11:37:51.056540 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:37:51.056513    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-66b9664c88slmlh" podUID="a33d5fe5-3633-466f-888d-5250a697bed0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.34:8001/health\": dial tcp 10.133.0.34:8001: connect: connection refused"
May 25 11:37:51.063586 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:37:51.063561    2576 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-cdng2p5"
May 25 11:37:51.063679 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:37:51.063619    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-cdng2p5"
May 25 11:37:51.064976 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:37:51.064948    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-cdng2p5" podUID="599e3fe7-d07d-405b-820a-42e866f1636d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/health\": dial tcp 10.133.0.35:8000: connect: connection refused"
May 25 11:38:01.054893 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:38:01.054843    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-66b9664c88slmlh" podUID="a33d5fe5-3633-466f-888d-5250a697bed0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.34:8001/health\": dial tcp 10.133.0.34:8001: connect: connection refused"
May 25 11:38:01.063929 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:38:01.063870    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-cdng2p5" podUID="599e3fe7-d07d-405b-820a-42e866f1636d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/health\": dial tcp 10.133.0.35:8000: connect: connection refused"
May 25 11:38:01.072169 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:38:01.072148    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-66b9664c88slmlh"
May 25 11:38:06.701033 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:38:06.701008    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-test-kserve-cc6bc7969-2nj2j_06d3b63c-4efc-4afd-8ae1-ed31f1559f71/main/0.log"
May 25 11:38:06.701451 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:38:06.701419    2576 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-cc6bc7969-2nj2j"
May 25 11:38:06.758429 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:38:06.758396    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06d3b63c-4efc-4afd-8ae1-ed31f1559f71-dshm\") pod \"06d3b63c-4efc-4afd-8ae1-ed31f1559f71\" (UID: \"06d3b63c-4efc-4afd-8ae1-ed31f1559f71\") "
May 25 11:38:06.758576 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:38:06.758506    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06d3b63c-4efc-4afd-8ae1-ed31f1559f71-tmp-dir\") pod \"06d3b63c-4efc-4afd-8ae1-ed31f1559f71\" (UID: \"06d3b63c-4efc-4afd-8ae1-ed31f1559f71\") "
May 25 11:38:06.758576 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:38:06.758545    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06d3b63c-4efc-4afd-8ae1-ed31f1559f71-home\") pod \"06d3b63c-4efc-4afd-8ae1-ed31f1559f71\" (UID: \"06d3b63c-4efc-4afd-8ae1-ed31f1559f71\") "
May 25 11:38:06.758691 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:38:06.758576    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sbhjc\" (UniqueName: \"kubernetes.io/projected/06d3b63c-4efc-4afd-8ae1-ed31f1559f71-kube-api-access-sbhjc\") pod \"06d3b63c-4efc-4afd-8ae1-ed31f1559f71\" (UID: \"06d3b63c-4efc-4afd-8ae1-ed31f1559f71\") "
May 25 11:38:06.758691 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:38:06.758602    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/06d3b63c-4efc-4afd-8ae1-ed31f1559f71-kserve-provision-location\") pod \"06d3b63c-4efc-4afd-8ae1-ed31f1559f71\" (UID: \"06d3b63c-4efc-4afd-8ae1-ed31f1559f71\") "
May 25 11:38:06.758691 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:38:06.758642    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06d3b63c-4efc-4afd-8ae1-ed31f1559f71-model-cache\") pod \"06d3b63c-4efc-4afd-8ae1-ed31f1559f71\" (UID: \"06d3b63c-4efc-4afd-8ae1-ed31f1559f71\") "
May 25 11:38:06.758858 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:38:06.758731    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06d3b63c-4efc-4afd-8ae1-ed31f1559f71-tls-certs\") pod \"06d3b63c-4efc-4afd-8ae1-ed31f1559f71\" (UID: \"06d3b63c-4efc-4afd-8ae1-ed31f1559f71\") "
May 25 11:38:06.759135 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:38:06.759103    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06d3b63c-4efc-4afd-8ae1-ed31f1559f71-model-cache" (OuterVolumeSpecName: "model-cache") pod "06d3b63c-4efc-4afd-8ae1-ed31f1559f71" (UID: "06d3b63c-4efc-4afd-8ae1-ed31f1559f71"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:38:06.759701 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:38:06.759677    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06d3b63c-4efc-4afd-8ae1-ed31f1559f71-home" (OuterVolumeSpecName: "home") pod "06d3b63c-4efc-4afd-8ae1-ed31f1559f71" (UID: "06d3b63c-4efc-4afd-8ae1-ed31f1559f71"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:38:06.761168 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:38:06.761006    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06d3b63c-4efc-4afd-8ae1-ed31f1559f71-dshm" (OuterVolumeSpecName: "dshm") pod "06d3b63c-4efc-4afd-8ae1-ed31f1559f71" (UID: "06d3b63c-4efc-4afd-8ae1-ed31f1559f71"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:38:06.761168 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:38:06.761188    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06d3b63c-4efc-4afd-8ae1-ed31f1559f71-kube-api-access-sbhjc" (OuterVolumeSpecName: "kube-api-access-sbhjc") pod "06d3b63c-4efc-4afd-8ae1-ed31f1559f71" (UID: "06d3b63c-4efc-4afd-8ae1-ed31f1559f71"). InnerVolumeSpecName "kube-api-access-sbhjc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 11:38:06.761646 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:38:06.761620    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06d3b63c-4efc-4afd-8ae1-ed31f1559f71-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "06d3b63c-4efc-4afd-8ae1-ed31f1559f71" (UID: "06d3b63c-4efc-4afd-8ae1-ed31f1559f71"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 11:38:06.771184 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:38:06.771160    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06d3b63c-4efc-4afd-8ae1-ed31f1559f71-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "06d3b63c-4efc-4afd-8ae1-ed31f1559f71" (UID: "06d3b63c-4efc-4afd-8ae1-ed31f1559f71"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:38:06.828536 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:38:06.828497    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06d3b63c-4efc-4afd-8ae1-ed31f1559f71-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "06d3b63c-4efc-4afd-8ae1-ed31f1559f71" (UID: "06d3b63c-4efc-4afd-8ae1-ed31f1559f71"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:38:06.860599 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:38:06.860557    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06d3b63c-4efc-4afd-8ae1-ed31f1559f71-tmp-dir\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
May 25 11:38:06.860599 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:38:06.860600    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06d3b63c-4efc-4afd-8ae1-ed31f1559f71-home\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
May 25 11:38:06.860767 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:38:06.860617    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sbhjc\" (UniqueName: \"kubernetes.io/projected/06d3b63c-4efc-4afd-8ae1-ed31f1559f71-kube-api-access-sbhjc\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
May 25 11:38:06.860767 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:38:06.860633    2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/06d3b63c-4efc-4afd-8ae1-ed31f1559f71-kserve-provision-location\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
May 25 11:38:06.860767 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:38:06.860649    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06d3b63c-4efc-4afd-8ae1-ed31f1559f71-model-cache\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
May 25 11:38:06.860767 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:38:06.860662    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06d3b63c-4efc-4afd-8ae1-ed31f1559f71-tls-certs\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
May 25 11:38:06.860767 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:38:06.860675    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06d3b63c-4efc-4afd-8ae1-ed31f1559f71-dshm\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
May 25 11:38:07.252808 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:38:07.252783    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-test-kserve-cc6bc7969-2nj2j_06d3b63c-4efc-4afd-8ae1-ed31f1559f71/main/0.log"
May 25 11:38:07.253199 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:38:07.253170    2576 generic.go:358] "Generic (PLEG): container finished" podID="06d3b63c-4efc-4afd-8ae1-ed31f1559f71" containerID="2b664f44b1ea8e48f207588fd81da6f1c85bddda2a4a4641ae3f9a921467f193" exitCode=137
May 25 11:38:07.253326 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:38:07.253245    2576 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-cc6bc7969-2nj2j"
May 25 11:38:07.253326 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:38:07.253263    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-cc6bc7969-2nj2j" event={"ID":"06d3b63c-4efc-4afd-8ae1-ed31f1559f71","Type":"ContainerDied","Data":"2b664f44b1ea8e48f207588fd81da6f1c85bddda2a4a4641ae3f9a921467f193"}
May 25 11:38:07.253425 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:38:07.253325    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-cc6bc7969-2nj2j" event={"ID":"06d3b63c-4efc-4afd-8ae1-ed31f1559f71","Type":"ContainerDied","Data":"bb218ced20e29b616a018e6bbeff077483e9ee17909760416e2d0765148540eb"}
May 25 11:38:07.253425 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:38:07.253348    2576 scope.go:117] "RemoveContainer" containerID="2b664f44b1ea8e48f207588fd81da6f1c85bddda2a4a4641ae3f9a921467f193"
May 25 11:38:07.261933 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:38:07.261918    2576 scope.go:117] "RemoveContainer" containerID="a4fa78b1c647f824d94d03a10f9aee72ea26c27b29c305d928361ae101039c9f"
May 25 11:38:07.272140 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:38:07.272105    2576 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-cc6bc7969-2nj2j"]
May 25 11:38:07.278345 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:38:07.278318    2576 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-cc6bc7969-2nj2j"]
May 25 11:38:07.326099 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:38:07.326069    2576 scope.go:117] "RemoveContainer" containerID="2b664f44b1ea8e48f207588fd81da6f1c85bddda2a4a4641ae3f9a921467f193"
May 25 11:38:07.326500 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:38:07.326480    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b664f44b1ea8e48f207588fd81da6f1c85bddda2a4a4641ae3f9a921467f193\": container with ID starting with 2b664f44b1ea8e48f207588fd81da6f1c85bddda2a4a4641ae3f9a921467f193 not found: ID does not exist" containerID="2b664f44b1ea8e48f207588fd81da6f1c85bddda2a4a4641ae3f9a921467f193"
May 25 11:38:07.326577 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:38:07.326508    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b664f44b1ea8e48f207588fd81da6f1c85bddda2a4a4641ae3f9a921467f193"} err="failed to get container status \"2b664f44b1ea8e48f207588fd81da6f1c85bddda2a4a4641ae3f9a921467f193\": rpc error: code = NotFound desc = could not find container \"2b664f44b1ea8e48f207588fd81da6f1c85bddda2a4a4641ae3f9a921467f193\": container with ID starting with 2b664f44b1ea8e48f207588fd81da6f1c85bddda2a4a4641ae3f9a921467f193 not found: ID does not exist"
May 25 11:38:07.326577 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:38:07.326531    2576 scope.go:117] "RemoveContainer" containerID="a4fa78b1c647f824d94d03a10f9aee72ea26c27b29c305d928361ae101039c9f"
May 25 11:38:07.326839 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:38:07.326807    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4fa78b1c647f824d94d03a10f9aee72ea26c27b29c305d928361ae101039c9f\": container with ID starting with a4fa78b1c647f824d94d03a10f9aee72ea26c27b29c305d928361ae101039c9f not found: ID does not exist" containerID="a4fa78b1c647f824d94d03a10f9aee72ea26c27b29c305d928361ae101039c9f"
May 25 11:38:07.326941 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:38:07.326850    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4fa78b1c647f824d94d03a10f9aee72ea26c27b29c305d928361ae101039c9f"} err="failed to get container status \"a4fa78b1c647f824d94d03a10f9aee72ea26c27b29c305d928361ae101039c9f\": rpc error: code = NotFound desc = could not find container \"a4fa78b1c647f824d94d03a10f9aee72ea26c27b29c305d928361ae101039c9f\": container with ID starting with a4fa78b1c647f824d94d03a10f9aee72ea26c27b29c305d928361ae101039c9f not found: ID does not exist"
May 25 11:38:09.100472 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:38:09.100437    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="06d3b63c-4efc-4afd-8ae1-ed31f1559f71" path="/var/lib/kubelet/pods/06d3b63c-4efc-4afd-8ae1-ed31f1559f71/volumes"
May 25 11:38:11.055205 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:38:11.055161    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-66b9664c88slmlh" podUID="a33d5fe5-3633-466f-888d-5250a697bed0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.34:8001/health\": dial tcp 10.133.0.34:8001: connect: connection refused"
May 25 11:38:11.063897 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:38:11.063858    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-cdng2p5" podUID="599e3fe7-d07d-405b-820a-42e866f1636d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/health\": dial tcp 10.133.0.35:8000: connect: connection refused"
May 25 11:38:21.055233 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:38:21.055186    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-66b9664c88slmlh" podUID="a33d5fe5-3633-466f-888d-5250a697bed0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.34:8001/health\": dial tcp 10.133.0.34:8001: connect: connection refused"
May 25 11:38:21.063502 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:38:21.063467    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-cdng2p5" podUID="599e3fe7-d07d-405b-820a-42e866f1636d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/health\": dial tcp 10.133.0.35:8000: connect: connection refused"
May 25 11:38:31.055054 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:38:31.054973    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-66b9664c88slmlh" podUID="a33d5fe5-3633-466f-888d-5250a697bed0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.34:8001/health\": dial tcp 10.133.0.34:8001: connect: connection refused"
May 25 11:38:31.063877 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:38:31.063847    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-cdng2p5" podUID="599e3fe7-d07d-405b-820a-42e866f1636d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/health\": dial tcp 10.133.0.35:8000: connect: connection refused"
May 25 11:38:41.054620 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:38:41.054553    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-66b9664c88slmlh" podUID="a33d5fe5-3633-466f-888d-5250a697bed0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.34:8001/health\": dial tcp 10.133.0.34:8001: connect: connection refused"
May 25 11:38:41.063257 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:38:41.063213    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-cdng2p5" podUID="599e3fe7-d07d-405b-820a-42e866f1636d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/health\": dial tcp 10.133.0.35:8000: connect: connection refused"
May 25 11:38:51.054941 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:38:51.054882    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-66b9664c88slmlh" podUID="a33d5fe5-3633-466f-888d-5250a697bed0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.34:8001/health\": dial tcp 10.133.0.34:8001: connect: connection refused"
May 25 11:38:51.063032 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:38:51.063002    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-cdng2p5" podUID="599e3fe7-d07d-405b-820a-42e866f1636d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/health\": dial tcp 10.133.0.35:8000: connect: connection refused"
May 25 11:39:01.054797 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:39:01.054749    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-66b9664c88slmlh" podUID="a33d5fe5-3633-466f-888d-5250a697bed0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.34:8001/health\": dial tcp 10.133.0.34:8001: connect: connection refused"
May 25 11:39:01.063162 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:39:01.063124    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-cdng2p5" podUID="599e3fe7-d07d-405b-820a-42e866f1636d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/health\": dial tcp 10.133.0.35:8000: connect: connection refused"
May 25 11:39:11.054352 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:39:11.054310    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-66b9664c88slmlh" podUID="a33d5fe5-3633-466f-888d-5250a697bed0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.34:8001/health\": dial tcp 10.133.0.34:8001: connect: connection refused"
May 25 11:39:11.063082 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:39:11.063044    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-cdng2p5" podUID="599e3fe7-d07d-405b-820a-42e866f1636d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/health\": dial tcp 10.133.0.35:8000: connect: connection refused"
May 25 11:39:21.054592 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:39:21.054539    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-66b9664c88slmlh" podUID="a33d5fe5-3633-466f-888d-5250a697bed0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.34:8001/health\": dial tcp 10.133.0.34:8001: connect: connection refused"
May 25 11:39:21.063883 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:39:21.063845    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-cdng2p5" podUID="599e3fe7-d07d-405b-820a-42e866f1636d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/health\": dial tcp 10.133.0.35:8000: connect: connection refused"
May 25 11:39:31.054782 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:39:31.054730    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-66b9664c88slmlh" podUID="a33d5fe5-3633-466f-888d-5250a697bed0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.34:8001/health\": dial tcp 10.133.0.34:8001: connect: connection refused"
May 25 11:39:31.063550 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:39:31.063518    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-cdng2p5" podUID="599e3fe7-d07d-405b-820a-42e866f1636d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/health\": dial tcp 10.133.0.35:8000: connect: connection refused"
May 25 11:39:41.055063 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:39:41.055020    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-66b9664c88slmlh" podUID="a33d5fe5-3633-466f-888d-5250a697bed0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.34:8001/health\": dial tcp 10.133.0.34:8001: connect: connection refused"
May 25 11:39:41.062917 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:39:41.062890    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-cdng2p5" podUID="599e3fe7-d07d-405b-820a-42e866f1636d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/health\": dial tcp 10.133.0.35:8000: connect: connection refused"
May 25 11:39:51.054301 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:39:51.054248    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-66b9664c88slmlh" podUID="a33d5fe5-3633-466f-888d-5250a697bed0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.34:8001/health\": dial tcp 10.133.0.34:8001: connect: connection refused"
May 25 11:39:51.063340 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:39:51.063305    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-cdng2p5" podUID="599e3fe7-d07d-405b-820a-42e866f1636d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/health\": dial tcp 10.133.0.35:8000: connect: connection refused"
May 25 11:40:01.071732 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:40:01.071701    2576 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-66b9664c88slmlh"
May 25 11:40:01.075235 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:40:01.075210    2576 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-cdng2p5"
May 25 11:40:01.087786 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:40:01.087744    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-cdng2p5"
May 25 11:40:01.088166 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:40:01.088149    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-66b9664c88slmlh"
May 25 11:40:08.988534 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:40:08.988491    2576 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-cdng2p5"]
May 25 11:40:08.988969 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:40:08.988913    2576 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-cdng2p5" podUID="599e3fe7-d07d-405b-820a-42e866f1636d" containerName="main" containerID="cri-o://7e4b640b0cf7db34fa537cb82fbc2f323e65ed91006c262cc7e0b84a2148d0c1" gracePeriod=30
May 25 11:40:08.992821 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:40:08.992798    2576 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-66b9664c88slmlh"]
May 25 11:40:08.993196 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:40:08.993173    2576 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-66b9664c88slmlh" podUID="a33d5fe5-3633-466f-888d-5250a697bed0" containerName="main" containerID="cri-o://d2dfed07f97d309cb9295959a696a162ecbf0c37c84deb20a02c172733ed5fc5" gracePeriod=30
May 25 11:40:23.479421 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:40:23.479386    2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-545854866-cq97p"]
May 25 11:40:23.479869 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:40:23.479749    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="06d3b63c-4efc-4afd-8ae1-ed31f1559f71" containerName="main"
May 25 11:40:23.479869 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:40:23.479761    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="06d3b63c-4efc-4afd-8ae1-ed31f1559f71" containerName="main"
May 25 11:40:23.479869 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:40:23.479786    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="06d3b63c-4efc-4afd-8ae1-ed31f1559f71" containerName="storage-initializer"
May 25 11:40:23.479869 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:40:23.479794    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="06d3b63c-4efc-4afd-8ae1-ed31f1559f71" containerName="storage-initializer"
May 25 11:40:23.479869 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:40:23.479844    2576 memory_manager.go:356] "RemoveStaleState removing state" podUID="06d3b63c-4efc-4afd-8ae1-ed31f1559f71" containerName="main"
May 25 11:40:23.484328 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:40:23.484307    2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6b47c648bc-kgnng"]
May 25 11:40:23.484462 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:40:23.484443    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-545854866-cq97p"
May 25 11:40:23.487645 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:40:23.487631    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6b47c648bc-kgnng"
May 25 11:40:23.493227 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:40:23.493211    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"custom-route-timeout-pd-test-kserve-dockercfg-68v4v\""
May 25 11:40:23.494847 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:40:23.494830    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"custom-route-timeout-pd-test-kserve-self-signed-certs\""
May 25 11:40:23.514766 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:40:23.514740    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6b47c648bc-kgnng"]
May 25 11:40:23.522128 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:40:23.522107    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-545854866-cq97p"]
May 25 11:40:23.637872 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:40:23.637848    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/77cb6ea6-0019-4f5a-a219-72854cf6c824-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-prefill-545854866-cq97p\" (UID: \"77cb6ea6-0019-4f5a-a219-72854cf6c824\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-545854866-cq97p"
May 25 11:40:23.638006 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:40:23.637877    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/77cb6ea6-0019-4f5a-a219-72854cf6c824-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-prefill-545854866-cq97p\" (UID: \"77cb6ea6-0019-4f5a-a219-72854cf6c824\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-545854866-cq97p"
May 25 11:40:23.638006 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:40:23.637903    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/77cb6ea6-0019-4f5a-a219-72854cf6c824-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-prefill-545854866-cq97p\" (UID: \"77cb6ea6-0019-4f5a-a219-72854cf6c824\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-545854866-cq97p"
May 25 11:40:23.638006 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:40:23.637958    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmgkb\" (UniqueName: \"kubernetes.io/projected/77cb6ea6-0019-4f5a-a219-72854cf6c824-kube-api-access-tmgkb\") pod \"custom-route-timeout-pd-test-kserve-prefill-545854866-cq97p\" (UID: \"77cb6ea6-0019-4f5a-a219-72854cf6c824\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-545854866-cq97p"
May 25 11:40:23.638006 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:40:23.637984    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c0dfef3-38b4-4195-a07c-acab5c17e850-home\") pod \"custom-route-timeout-pd-test-kserve-6b47c648bc-kgnng\" (UID: \"1c0dfef3-38b4-4195-a07c-acab5c17e850\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6b47c648bc-kgnng"
May 25 11:40:23.638157 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:40:23.638026    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c0dfef3-38b4-4195-a07c-acab5c17e850-model-cache\") pod \"custom-route-timeout-pd-test-kserve-6b47c648bc-kgnng\" (UID: \"1c0dfef3-38b4-4195-a07c-acab5c17e850\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6b47c648bc-kgnng"
May 25 11:40:23.638157 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:40:23.638048    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/77cb6ea6-0019-4f5a-a219-72854cf6c824-model-cache\") pod \"custom-route-timeout-pd-test-kserve-prefill-545854866-cq97p\" (UID: \"77cb6ea6-0019-4f5a-a219-72854cf6c824\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-545854866-cq97p"
May 25 11:40:23.638157 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:40:23.638068    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c0dfef3-38b4-4195-a07c-acab5c17e850-dshm\") pod \"custom-route-timeout-pd-test-kserve-6b47c648bc-kgnng\" (UID: \"1c0dfef3-38b4-4195-a07c-acab5c17e850\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6b47c648bc-kgnng"
May 25 11:40:23.638157 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:40:23.638139    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/77cb6ea6-0019-4f5a-a219-72854cf6c824-dshm\") pod \"custom-route-timeout-pd-test-kserve-prefill-545854866-cq97p\" (UID: \"77cb6ea6-0019-4f5a-a219-72854cf6c824\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-545854866-cq97p"
May 25 11:40:23.638317 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:40:23.638165    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c0dfef3-38b4-4195-a07c-acab5c17e850-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-6b47c648bc-kgnng\" (UID: \"1c0dfef3-38b4-4195-a07c-acab5c17e850\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6b47c648bc-kgnng"
May 25 11:40:23.638317 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:40:23.638191    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c0dfef3-38b4-4195-a07c-acab5c17e850-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-6b47c648bc-kgnng\" (UID: \"1c0dfef3-38b4-4195-a07c-acab5c17e850\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6b47c648bc-kgnng"
May 25 11:40:23.638317 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:40:23.638213    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1c0dfef3-38b4-4195-a07c-acab5c17e850-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-6b47c648bc-kgnng\" (UID: \"1c0dfef3-38b4-4195-a07c-acab5c17e850\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6b47c648bc-kgnng"
May 25 11:40:23.638317 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:40:23.638235    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/77cb6ea6-0019-4f5a-a219-72854cf6c824-home\") pod \"custom-route-timeout-pd-test-kserve-prefill-545854866-cq97p\" (UID: \"77cb6ea6-0019-4f5a-a219-72854cf6c824\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-545854866-cq97p"
May 25 11:40:23.638317 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:40:23.638263    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2hww\" (UniqueName: \"kubernetes.io/projected/1c0dfef3-38b4-4195-a07c-acab5c17e850-kube-api-access-r2hww\") pod \"custom-route-timeout-pd-test-kserve-6b47c648bc-kgnng\" (UID: \"1c0dfef3-38b4-4195-a07c-acab5c17e850\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6b47c648bc-kgnng"
May 25 11:40:23.739177 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:40:23.739111    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c0dfef3-38b4-4195-a07c-acab5c17e850-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-6b47c648bc-kgnng\" (UID: \"1c0dfef3-38b4-4195-a07c-acab5c17e850\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6b47c648bc-kgnng"
May 25 11:40:23.739177 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:40:23.739149    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1c0dfef3-38b4-4195-a07c-acab5c17e850-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-6b47c648bc-kgnng\" (UID: \"1c0dfef3-38b4-4195-a07c-acab5c17e850\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6b47c648bc-kgnng"
May 25 11:40:23.739177 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:40:23.739174    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/77cb6ea6-0019-4f5a-a219-72854cf6c824-home\") pod \"custom-route-timeout-pd-test-kserve-prefill-545854866-cq97p\" (UID: \"77cb6ea6-0019-4f5a-a219-72854cf6c824\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-545854866-cq97p"
May 25 11:40:23.739388 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:40:23.739208    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-r2hww\" (UniqueName: \"kubernetes.io/projected/1c0dfef3-38b4-4195-a07c-acab5c17e850-kube-api-access-r2hww\") pod \"custom-route-timeout-pd-test-kserve-6b47c648bc-kgnng\" (UID: \"1c0dfef3-38b4-4195-a07c-acab5c17e850\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6b47c648bc-kgnng"
May 25 11:40:23.739388 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:40:23.739251    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/77cb6ea6-0019-4f5a-a219-72854cf6c824-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-prefill-545854866-cq97p\" (UID: \"77cb6ea6-0019-4f5a-a219-72854cf6c824\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-545854866-cq97p"
May 25 11:40:23.739388 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:40:23.739369    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/77cb6ea6-0019-4f5a-a219-72854cf6c824-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-prefill-545854866-cq97p\" (UID: \"77cb6ea6-0019-4f5a-a219-72854cf6c824\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-545854866-cq97p"
May 25 11:40:23.739535 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:40:23.739412    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/77cb6ea6-0019-4f5a-a219-72854cf6c824-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-prefill-545854866-cq97p\" (UID: \"77cb6ea6-0019-4f5a-a219-72854cf6c824\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-545854866-cq97p"
May 25 11:40:23.739535 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:40:23.739481    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c0dfef3-38b4-4195-a07c-acab5c17e850-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-6b47c648bc-kgnng\" (UID: \"1c0dfef3-38b4-4195-a07c-acab5c17e850\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6b47c648bc-kgnng"
May 25 11:40:23.739650 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:40:23.739539    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-tmgkb\" (UniqueName: \"kubernetes.io/projected/77cb6ea6-0019-4f5a-a219-72854cf6c824-kube-api-access-tmgkb\") pod \"custom-route-timeout-pd-test-kserve-prefill-545854866-cq97p\" (UID: \"77cb6ea6-0019-4f5a-a219-72854cf6c824\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-545854866-cq97p"
May 25 11:40:23.739650 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:40:23.739577    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c0dfef3-38b4-4195-a07c-acab5c17e850-home\") pod \"custom-route-timeout-pd-test-kserve-6b47c648bc-kgnng\" (UID: \"1c0dfef3-38b4-4195-a07c-acab5c17e850\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6b47c648bc-kgnng"
May 25 11:40:23.739650 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:40:23.739538    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1c0dfef3-38b4-4195-a07c-acab5c17e850-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-6b47c648bc-kgnng\" (UID: \"1c0dfef3-38b4-4195-a07c-acab5c17e850\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6b47c648bc-kgnng"
May 25 11:40:23.739650 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:40:23.739639    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c0dfef3-38b4-4195-a07c-acab5c17e850-model-cache\") pod \"custom-route-timeout-pd-test-kserve-6b47c648bc-kgnng\" (UID: \"1c0dfef3-38b4-4195-a07c-acab5c17e850\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6b47c648bc-kgnng"
May 25 11:40:23.739650 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:40:23.739643    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/77cb6ea6-0019-4f5a-a219-72854cf6c824-home\") pod \"custom-route-timeout-pd-test-kserve-prefill-545854866-cq97p\" (UID: \"77cb6ea6-0019-4f5a-a219-72854cf6c824\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-545854866-cq97p"
May 25 11:40:23.739889 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:40:23.739680    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/77cb6ea6-0019-4f5a-a219-72854cf6c824-model-cache\") pod \"custom-route-timeout-pd-test-kserve-prefill-545854866-cq97p\" (UID: \"77cb6ea6-0019-4f5a-a219-72854cf6c824\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-545854866-cq97p"
May 25 11:40:23.739889 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:40:23.739709    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c0dfef3-38b4-4195-a07c-acab5c17e850-dshm\") pod \"custom-route-timeout-pd-test-kserve-6b47c648bc-kgnng\" (UID: \"1c0dfef3-38b4-4195-a07c-acab5c17e850\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6b47c648bc-kgnng"
May 25 11:40:23.739889 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:40:23.739761    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/77cb6ea6-0019-4f5a-a219-72854cf6c824-dshm\") pod \"custom-route-timeout-pd-test-kserve-prefill-545854866-cq97p\" (UID: \"77cb6ea6-0019-4f5a-a219-72854cf6c824\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-545854866-cq97p"
May 25 11:40:23.739889 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:40:23.739788    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/77cb6ea6-0019-4f5a-a219-72854cf6c824-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-prefill-545854866-cq97p\" (UID: \"77cb6ea6-0019-4f5a-a219-72854cf6c824\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-545854866-cq97p"
May 25 11:40:23.739889 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:40:23.739789    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c0dfef3-38b4-4195-a07c-acab5c17e850-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-6b47c648bc-kgnng\" (UID: \"1c0dfef3-38b4-4195-a07c-acab5c17e850\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6b47c648bc-kgnng"
May 25 11:40:23.739889 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:40:23.739838    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/77cb6ea6-0019-4f5a-a219-72854cf6c824-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-prefill-545854866-cq97p\" (UID: \"77cb6ea6-0019-4f5a-a219-72854cf6c824\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-545854866-cq97p"
May 25 11:40:23.740246 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:40:23.739911    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c0dfef3-38b4-4195-a07c-acab5c17e850-home\") pod \"custom-route-timeout-pd-test-kserve-6b47c648bc-kgnng\" (UID: \"1c0dfef3-38b4-4195-a07c-acab5c17e850\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6b47c648bc-kgnng"
May 25 11:40:23.740246 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:40:23.739980    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c0dfef3-38b4-4195-a07c-acab5c17e850-model-cache\") pod \"custom-route-timeout-pd-test-kserve-6b47c648bc-kgnng\" (UID: \"1c0dfef3-38b4-4195-a07c-acab5c17e850\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6b47c648bc-kgnng"
May 25 11:40:23.740246 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:40:23.739990    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/77cb6ea6-0019-4f5a-a219-72854cf6c824-model-cache\") pod \"custom-route-timeout-pd-test-kserve-prefill-545854866-cq97p\" (UID: \"77cb6ea6-0019-4f5a-a219-72854cf6c824\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-545854866-cq97p"
May 25 11:40:23.741979 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:40:23.741959    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c0dfef3-38b4-4195-a07c-acab5c17e850-dshm\") pod \"custom-route-timeout-pd-test-kserve-6b47c648bc-kgnng\" (UID: \"1c0dfef3-38b4-4195-a07c-acab5c17e850\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6b47c648bc-kgnng"
May 25 11:40:23.742091 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:40:23.742032    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/77cb6ea6-0019-4f5a-a219-72854cf6c824-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-prefill-545854866-cq97p\" (UID: \"77cb6ea6-0019-4f5a-a219-72854cf6c824\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-545854866-cq97p"
May 25 11:40:23.742091 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:40:23.742038    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/77cb6ea6-0019-4f5a-a219-72854cf6c824-dshm\") pod \"custom-route-timeout-pd-test-kserve-prefill-545854866-cq97p\" (UID: \"77cb6ea6-0019-4f5a-a219-72854cf6c824\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-545854866-cq97p"
May 25 11:40:23.742459 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:40:23.742433    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c0dfef3-38b4-4195-a07c-acab5c17e850-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-6b47c648bc-kgnng\" (UID: \"1c0dfef3-38b4-4195-a07c-acab5c17e850\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6b47c648bc-kgnng"
May 25 11:40:23.747436 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:40:23.747410    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmgkb\" (UniqueName: \"kubernetes.io/projected/77cb6ea6-0019-4f5a-a219-72854cf6c824-kube-api-access-tmgkb\") pod \"custom-route-timeout-pd-test-kserve-prefill-545854866-cq97p\" (UID: \"77cb6ea6-0019-4f5a-a219-72854cf6c824\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-545854866-cq97p"
May 25 11:40:23.747511 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:40:23.747440    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2hww\" (UniqueName: \"kubernetes.io/projected/1c0dfef3-38b4-4195-a07c-acab5c17e850-kube-api-access-r2hww\") pod \"custom-route-timeout-pd-test-kserve-6b47c648bc-kgnng\" (UID: \"1c0dfef3-38b4-4195-a07c-acab5c17e850\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6b47c648bc-kgnng"
May 25 11:40:23.795192 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:40:23.795169    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-545854866-cq97p"
May 25 11:40:23.799867 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:40:23.799849    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6b47c648bc-kgnng"
May 25 11:40:23.936881 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:40:23.936853    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-545854866-cq97p"]
May 25 11:40:23.939151 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:40:23.939119    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77cb6ea6_0019_4f5a_a219_72854cf6c824.slice/crio-8dc7123b3b49fa7e5031616fb63543de87d637742b515ed81be4583e0cdd5c1f WatchSource:0}: Error finding container 8dc7123b3b49fa7e5031616fb63543de87d637742b515ed81be4583e0cdd5c1f: Status 404 returned error can't find the container with id 8dc7123b3b49fa7e5031616fb63543de87d637742b515ed81be4583e0cdd5c1f
May 25 11:40:23.953610 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:40:23.953582    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6b47c648bc-kgnng"]
May 25 11:40:23.955861 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:40:23.955838    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c0dfef3_38b4_4195_a07c_acab5c17e850.slice/crio-564a5ea63cd4e8a3306ea79ab8bce9ae9f3586f2591f432de4c3f59f0b632bc2 WatchSource:0}: Error finding container 564a5ea63cd4e8a3306ea79ab8bce9ae9f3586f2591f432de4c3f59f0b632bc2: Status 404 returned error can't find the container with id 564a5ea63cd4e8a3306ea79ab8bce9ae9f3586f2591f432de4c3f59f0b632bc2
May 25 11:40:24.749687 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:40:24.749652    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-545854866-cq97p" event={"ID":"77cb6ea6-0019-4f5a-a219-72854cf6c824","Type":"ContainerStarted","Data":"96326c724e8dee4b11634d6fadda73fd76d303307bdccef68a13cf476a08c4c6"}
May 25 11:40:24.749687 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:40:24.749690    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-545854866-cq97p" event={"ID":"77cb6ea6-0019-4f5a-a219-72854cf6c824","Type":"ContainerStarted","Data":"8dc7123b3b49fa7e5031616fb63543de87d637742b515ed81be4583e0cdd5c1f"}
May 25 11:40:24.751225 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:40:24.751194    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6b47c648bc-kgnng" event={"ID":"1c0dfef3-38b4-4195-a07c-acab5c17e850","Type":"ContainerStarted","Data":"6b34ee9d8e732ca40b6f96fa3f02559f62b2f50b0c96d9b4e7da362fcce8abe0"}
May 25 11:40:24.751363 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:40:24.751233    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6b47c648bc-kgnng" event={"ID":"1c0dfef3-38b4-4195-a07c-acab5c17e850","Type":"ContainerStarted","Data":"564a5ea63cd4e8a3306ea79ab8bce9ae9f3586f2591f432de4c3f59f0b632bc2"}
May 25 11:40:24.751363 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:40:24.751313    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6b47c648bc-kgnng"
May 25 11:40:25.758621 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:40:25.758574    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6b47c648bc-kgnng" event={"ID":"1c0dfef3-38b4-4195-a07c-acab5c17e850","Type":"ContainerStarted","Data":"b3bef6c9e7c5f08232bbb889469fe7839c656f687bf098a8832a0944a5a2cd19"}
May 25 11:40:29.775137 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:40:29.775093    2576 generic.go:358] "Generic (PLEG): container finished" podID="77cb6ea6-0019-4f5a-a219-72854cf6c824" containerID="96326c724e8dee4b11634d6fadda73fd76d303307bdccef68a13cf476a08c4c6" exitCode=0
May 25 11:40:29.775715 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:40:29.775160    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-545854866-cq97p" event={"ID":"77cb6ea6-0019-4f5a-a219-72854cf6c824","Type":"ContainerDied","Data":"96326c724e8dee4b11634d6fadda73fd76d303307bdccef68a13cf476a08c4c6"}
May 25 11:40:29.777124 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:40:29.777101    2576 generic.go:358] "Generic (PLEG): container finished" podID="1c0dfef3-38b4-4195-a07c-acab5c17e850" containerID="b3bef6c9e7c5f08232bbb889469fe7839c656f687bf098a8832a0944a5a2cd19" exitCode=0
May 25 11:40:29.777224 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:40:29.777170    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6b47c648bc-kgnng" event={"ID":"1c0dfef3-38b4-4195-a07c-acab5c17e850","Type":"ContainerDied","Data":"b3bef6c9e7c5f08232bbb889469fe7839c656f687bf098a8832a0944a5a2cd19"}
May 25 11:40:30.784364 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:40:30.784317    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6b47c648bc-kgnng" event={"ID":"1c0dfef3-38b4-4195-a07c-acab5c17e850","Type":"ContainerStarted","Data":"4f8533625cf9d1f67bd9e991c79e026513dd35dbc43158603a589b1f2dc54fc9"}
May 25 11:40:30.786428 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:40:30.786389    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-545854866-cq97p" event={"ID":"77cb6ea6-0019-4f5a-a219-72854cf6c824","Type":"ContainerStarted","Data":"838bbab20ca93f7029560b5dfadda21e257c2acd4c1b1deb3d37563855a85dfd"}
May 25 11:40:30.811624 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:40:30.811572    2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6b47c648bc-kgnng" podStartSLOduration=7.811557701 podStartE2EDuration="7.811557701s" podCreationTimestamp="2026-05-25 11:40:23 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 11:40:30.809118191 +0000 UTC m=+1376.326849649" watchObservedRunningTime="2026-05-25 11:40:30.811557701 +0000 UTC m=+1376.329289157"
May 25 11:40:30.830883 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:40:30.830766    2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-545854866-cq97p" podStartSLOduration=7.830747494 podStartE2EDuration="7.830747494s" podCreationTimestamp="2026-05-25 11:40:23 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 11:40:30.829799314 +0000 UTC m=+1376.347530771" watchObservedRunningTime="2026-05-25 11:40:30.830747494 +0000 UTC m=+1376.348478952"
May 25 11:40:33.795529 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:40:33.795488    2576 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-545854866-cq97p"
May 25 11:40:33.796002 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:40:33.795541    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-545854866-cq97p"
May 25 11:40:33.797435 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:40:33.797400    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-545854866-cq97p" podUID="77cb6ea6-0019-4f5a-a219-72854cf6c824" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/health\": dial tcp 10.133.0.36:8000: connect: connection refused"
May 25 11:40:33.799914 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:40:33.799893    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6b47c648bc-kgnng"
May 25 11:40:33.800011 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:40:33.799933    2576 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6b47c648bc-kgnng"
May 25 11:40:33.801224 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:40:33.801191    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6b47c648bc-kgnng" podUID="1c0dfef3-38b4-4195-a07c-acab5c17e850" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8001/health\": dial tcp 10.133.0.37:8001: connect: connection refused"
May 25 11:40:33.819336 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:40:33.819314    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6b47c648bc-kgnng"
May 25 11:40:38.993483 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:40:38.993412    2576 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-66b9664c88slmlh" podUID="a33d5fe5-3633-466f-888d-5250a697bed0" containerName="llm-d-routing-sidecar" containerID="cri-o://ec8382e356e0a5f26fb5ad1ede5104af1e35c44b9c7116f1bf0b54f7079f7b23" gracePeriod=2
May 25 11:40:39.449389 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:40:39.449368    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-66b9664c88slmlh_a33d5fe5-3633-466f-888d-5250a697bed0/main/0.log"
May 25 11:40:39.450067 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:40:39.450047    2576 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-66b9664c88slmlh"
May 25 11:40:39.453926 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:40:39.453905    2576 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-cdng2p5"
May 25 11:40:39.501941 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:40:39.501912    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a33d5fe5-3633-466f-888d-5250a697bed0-tmp-dir\") pod \"a33d5fe5-3633-466f-888d-5250a697bed0\" (UID: \"a33d5fe5-3633-466f-888d-5250a697bed0\") "
May 25 11:40:39.502125 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:40:39.501949    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-24gzx\" (UniqueName: \"kubernetes.io/projected/a33d5fe5-3633-466f-888d-5250a697bed0-kube-api-access-24gzx\") pod \"a33d5fe5-3633-466f-888d-5250a697bed0\" (UID: \"a33d5fe5-3633-466f-888d-5250a697bed0\") "
May 25 11:40:39.502125 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:40:39.501985    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a33d5fe5-3633-466f-888d-5250a697bed0-model-cache\") pod \"a33d5fe5-3633-466f-888d-5250a697bed0\" (UID: \"a33d5fe5-3633-466f-888d-5250a697bed0\") "
May 25 11:40:39.502125 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:40:39.502037    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a33d5fe5-3633-466f-888d-5250a697bed0-tls-certs\") pod \"a33d5fe5-3633-466f-888d-5250a697bed0\" (UID: \"a33d5fe5-3633-466f-888d-5250a697bed0\") "
May 25 11:40:39.503249 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:40:39.502564    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a33d5fe5-3633-466f-888d-5250a697bed0-model-cache" (OuterVolumeSpecName: "model-cache") pod "a33d5fe5-3633-466f-888d-5250a697bed0" (UID: "a33d5fe5-3633-466f-888d-5250a697bed0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:40:39.503249 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:40:39.502591    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a33d5fe5-3633-466f-888d-5250a697bed0-home\") pod \"a33d5fe5-3633-466f-888d-5250a697bed0\" (UID: \"a33d5fe5-3633-466f-888d-5250a697bed0\") "
May 25 11:40:39.503249 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:40:39.502632    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a33d5fe5-3633-466f-888d-5250a697bed0-dshm\") pod \"a33d5fe5-3633-466f-888d-5250a697bed0\" (UID: \"a33d5fe5-3633-466f-888d-5250a697bed0\") "
May 25 11:40:39.503249 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:40:39.502682    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a33d5fe5-3633-466f-888d-5250a697bed0-kserve-provision-location\") pod \"a33d5fe5-3633-466f-888d-5250a697bed0\" (UID: \"a33d5fe5-3633-466f-888d-5250a697bed0\") "
May 25 11:40:39.503249 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:40:39.503030    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a33d5fe5-3633-466f-888d-5250a697bed0-model-cache\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
May 25 11:40:39.503249 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:40:39.503214    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a33d5fe5-3633-466f-888d-5250a697bed0-home" (OuterVolumeSpecName: "home") pod "a33d5fe5-3633-466f-888d-5250a697bed0" (UID: "a33d5fe5-3633-466f-888d-5250a697bed0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:40:39.505565 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:40:39.505539    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a33d5fe5-3633-466f-888d-5250a697bed0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a33d5fe5-3633-466f-888d-5250a697bed0" (UID: "a33d5fe5-3633-466f-888d-5250a697bed0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 11:40:39.505729 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:40:39.505667    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a33d5fe5-3633-466f-888d-5250a697bed0-dshm" (OuterVolumeSpecName: "dshm") pod "a33d5fe5-3633-466f-888d-5250a697bed0" (UID: "a33d5fe5-3633-466f-888d-5250a697bed0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:40:39.506143 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:40:39.506062    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a33d5fe5-3633-466f-888d-5250a697bed0-kube-api-access-24gzx" (OuterVolumeSpecName: "kube-api-access-24gzx") pod "a33d5fe5-3633-466f-888d-5250a697bed0" (UID: "a33d5fe5-3633-466f-888d-5250a697bed0"). InnerVolumeSpecName "kube-api-access-24gzx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 11:40:39.516946 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:40:39.516889    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a33d5fe5-3633-466f-888d-5250a697bed0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a33d5fe5-3633-466f-888d-5250a697bed0" (UID: "a33d5fe5-3633-466f-888d-5250a697bed0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:40:39.577116 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:40:39.577081    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a33d5fe5-3633-466f-888d-5250a697bed0-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a33d5fe5-3633-466f-888d-5250a697bed0" (UID: "a33d5fe5-3633-466f-888d-5250a697bed0"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:40:39.604233 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:40:39.604200    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/599e3fe7-d07d-405b-820a-42e866f1636d-model-cache\") pod \"599e3fe7-d07d-405b-820a-42e866f1636d\" (UID: \"599e3fe7-d07d-405b-820a-42e866f1636d\") "
May 25 11:40:39.604357 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:40:39.604290    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/599e3fe7-d07d-405b-820a-42e866f1636d-home\") pod \"599e3fe7-d07d-405b-820a-42e866f1636d\" (UID: \"599e3fe7-d07d-405b-820a-42e866f1636d\") "
May 25 11:40:39.604357 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:40:39.604325    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/599e3fe7-d07d-405b-820a-42e866f1636d-tls-certs\") pod \"599e3fe7-d07d-405b-820a-42e866f1636d\" (UID: \"599e3fe7-d07d-405b-820a-42e866f1636d\") "
May 25 11:40:39.604357 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:40:39.604344    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/599e3fe7-d07d-405b-820a-42e866f1636d-dshm\") pod \"599e3fe7-d07d-405b-820a-42e866f1636d\" (UID: \"599e3fe7-d07d-405b-820a-42e866f1636d\") "
May 25 11:40:39.604510 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:40:39.604363    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-26qwq\" (UniqueName: \"kubernetes.io/projected/599e3fe7-d07d-405b-820a-42e866f1636d-kube-api-access-26qwq\") pod \"599e3fe7-d07d-405b-820a-42e866f1636d\" (UID: \"599e3fe7-d07d-405b-820a-42e866f1636d\") "
May 25 11:40:39.604510 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:40:39.604429    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/599e3fe7-d07d-405b-820a-42e866f1636d-kserve-provision-location\") pod \"599e3fe7-d07d-405b-820a-42e866f1636d\" (UID: \"599e3fe7-d07d-405b-820a-42e866f1636d\") "
May 25 11:40:39.604510 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:40:39.604471    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/599e3fe7-d07d-405b-820a-42e866f1636d-tmp-dir\") pod \"599e3fe7-d07d-405b-820a-42e866f1636d\" (UID: \"599e3fe7-d07d-405b-820a-42e866f1636d\") "
May 25 11:40:39.604510 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:40:39.604491    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/599e3fe7-d07d-405b-820a-42e866f1636d-model-cache" (OuterVolumeSpecName: "model-cache") pod "599e3fe7-d07d-405b-820a-42e866f1636d" (UID: "599e3fe7-d07d-405b-820a-42e866f1636d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:40:39.605257 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:40:39.604757    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a33d5fe5-3633-466f-888d-5250a697bed0-tmp-dir\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
May 25 11:40:39.605257 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:40:39.604785    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/599e3fe7-d07d-405b-820a-42e866f1636d-model-cache\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
May 25 11:40:39.605257 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:40:39.604800    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-24gzx\" (UniqueName: \"kubernetes.io/projected/a33d5fe5-3633-466f-888d-5250a697bed0-kube-api-access-24gzx\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
May 25 11:40:39.605257 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:40:39.604815    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a33d5fe5-3633-466f-888d-5250a697bed0-tls-certs\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
May 25 11:40:39.605257 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:40:39.604830    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a33d5fe5-3633-466f-888d-5250a697bed0-home\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
May 25 11:40:39.605257 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:40:39.604844    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a33d5fe5-3633-466f-888d-5250a697bed0-dshm\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
May 25 11:40:39.605257 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:40:39.604857    2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a33d5fe5-3633-466f-888d-5250a697bed0-kserve-provision-location\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
May 25 11:40:39.605257 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:40:39.605221    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/599e3fe7-d07d-405b-820a-42e866f1636d-home" (OuterVolumeSpecName: "home") pod "599e3fe7-d07d-405b-820a-42e866f1636d" (UID: "599e3fe7-d07d-405b-820a-42e866f1636d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:40:39.607106 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:40:39.607081    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/599e3fe7-d07d-405b-820a-42e866f1636d-kube-api-access-26qwq" (OuterVolumeSpecName: "kube-api-access-26qwq") pod "599e3fe7-d07d-405b-820a-42e866f1636d" (UID: "599e3fe7-d07d-405b-820a-42e866f1636d"). InnerVolumeSpecName "kube-api-access-26qwq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 11:40:39.607202 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:40:39.607149    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/599e3fe7-d07d-405b-820a-42e866f1636d-dshm" (OuterVolumeSpecName: "dshm") pod "599e3fe7-d07d-405b-820a-42e866f1636d" (UID: "599e3fe7-d07d-405b-820a-42e866f1636d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:40:39.607259 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:40:39.607235    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/599e3fe7-d07d-405b-820a-42e866f1636d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "599e3fe7-d07d-405b-820a-42e866f1636d" (UID: "599e3fe7-d07d-405b-820a-42e866f1636d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 11:40:39.616540 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:40:39.616511    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/599e3fe7-d07d-405b-820a-42e866f1636d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "599e3fe7-d07d-405b-820a-42e866f1636d" (UID: "599e3fe7-d07d-405b-820a-42e866f1636d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:40:39.662306 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:40:39.662256    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/599e3fe7-d07d-405b-820a-42e866f1636d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "599e3fe7-d07d-405b-820a-42e866f1636d" (UID: "599e3fe7-d07d-405b-820a-42e866f1636d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:40:39.705349 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:40:39.705318    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/599e3fe7-d07d-405b-820a-42e866f1636d-home\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
May 25 11:40:39.705349 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:40:39.705347    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/599e3fe7-d07d-405b-820a-42e866f1636d-tls-certs\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
May 25 11:40:39.705505 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:40:39.705362    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/599e3fe7-d07d-405b-820a-42e866f1636d-dshm\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
May 25 11:40:39.705505 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:40:39.705375    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-26qwq\" (UniqueName: \"kubernetes.io/projected/599e3fe7-d07d-405b-820a-42e866f1636d-kube-api-access-26qwq\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
May 25 11:40:39.705505 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:40:39.705387    2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/599e3fe7-d07d-405b-820a-42e866f1636d-kserve-provision-location\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
May 25 11:40:39.705505 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:40:39.705400    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/599e3fe7-d07d-405b-820a-42e866f1636d-tmp-dir\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
May 25 11:40:39.823534 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:40:39.823452    2576 generic.go:358] "Generic (PLEG): container finished" podID="599e3fe7-d07d-405b-820a-42e866f1636d" containerID="7e4b640b0cf7db34fa537cb82fbc2f323e65ed91006c262cc7e0b84a2148d0c1" exitCode=137
May 25 11:40:39.823534 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:40:39.823526    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-cdng2p5" event={"ID":"599e3fe7-d07d-405b-820a-42e866f1636d","Type":"ContainerDied","Data":"7e4b640b0cf7db34fa537cb82fbc2f323e65ed91006c262cc7e0b84a2148d0c1"}
May 25 11:40:39.823748 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:40:39.823542    2576 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-cdng2p5"
May 25 11:40:39.823748 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:40:39.823561    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-cdng2p5" event={"ID":"599e3fe7-d07d-405b-820a-42e866f1636d","Type":"ContainerDied","Data":"2e2c0301a72e6ee32ed662a2348c27e88da92c2469df8bb43d6c73a3a74eb81a"}
May 25 11:40:39.823748 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:40:39.823582    2576 scope.go:117] "RemoveContainer" containerID="7e4b640b0cf7db34fa537cb82fbc2f323e65ed91006c262cc7e0b84a2148d0c1"
May 25 11:40:39.825023 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:40:39.825002    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-66b9664c88slmlh_a33d5fe5-3633-466f-888d-5250a697bed0/main/0.log"
May 25 11:40:39.825733 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:40:39.825709    2576 generic.go:358] "Generic (PLEG): container finished" podID="a33d5fe5-3633-466f-888d-5250a697bed0" containerID="d2dfed07f97d309cb9295959a696a162ecbf0c37c84deb20a02c172733ed5fc5" exitCode=137
May 25 11:40:39.825733 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:40:39.825733    2576 generic.go:358] "Generic (PLEG): container finished" podID="a33d5fe5-3633-466f-888d-5250a697bed0" containerID="ec8382e356e0a5f26fb5ad1ede5104af1e35c44b9c7116f1bf0b54f7079f7b23" exitCode=0
May 25 11:40:39.825893 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:40:39.825789    2576 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-66b9664c88slmlh"
May 25 11:40:39.825893 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:40:39.825794    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-66b9664c88slmlh" event={"ID":"a33d5fe5-3633-466f-888d-5250a697bed0","Type":"ContainerDied","Data":"d2dfed07f97d309cb9295959a696a162ecbf0c37c84deb20a02c172733ed5fc5"}
May 25 11:40:39.825893 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:40:39.825832    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-66b9664c88slmlh" event={"ID":"a33d5fe5-3633-466f-888d-5250a697bed0","Type":"ContainerDied","Data":"ec8382e356e0a5f26fb5ad1ede5104af1e35c44b9c7116f1bf0b54f7079f7b23"}
May 25 11:40:39.825893 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:40:39.825849    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-66b9664c88slmlh" event={"ID":"a33d5fe5-3633-466f-888d-5250a697bed0","Type":"ContainerDied","Data":"861282ee2ecf155c4ad5bfa8bcd1437cc482d5197582fead2cb5b70709118ce0"}
May 25 11:40:39.834440 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:40:39.834416    2576 scope.go:117] "RemoveContainer" containerID="cd56801d6efc44f41a3e07389b797c791b08761183bcb479bd6b383662dd435a"
May 25 11:40:39.855962 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:40:39.855904    2576 scope.go:117] "RemoveContainer" containerID="7e4b640b0cf7db34fa537cb82fbc2f323e65ed91006c262cc7e0b84a2148d0c1"
May 25 11:40:39.857122 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:40:39.856272    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e4b640b0cf7db34fa537cb82fbc2f323e65ed91006c262cc7e0b84a2148d0c1\": container with ID starting with 7e4b640b0cf7db34fa537cb82fbc2f323e65ed91006c262cc7e0b84a2148d0c1 not found: ID does not exist" containerID="7e4b640b0cf7db34fa537cb82fbc2f323e65ed91006c262cc7e0b84a2148d0c1"
May 25 11:40:39.857122 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:40:39.856326    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e4b640b0cf7db34fa537cb82fbc2f323e65ed91006c262cc7e0b84a2148d0c1"} err="failed to get container status \"7e4b640b0cf7db34fa537cb82fbc2f323e65ed91006c262cc7e0b84a2148d0c1\": rpc error: code = NotFound desc = could not find container \"7e4b640b0cf7db34fa537cb82fbc2f323e65ed91006c262cc7e0b84a2148d0c1\": container with ID starting with 7e4b640b0cf7db34fa537cb82fbc2f323e65ed91006c262cc7e0b84a2148d0c1 not found: ID does not exist"
May 25 11:40:39.857122 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:40:39.856351    2576 scope.go:117] "RemoveContainer" containerID="cd56801d6efc44f41a3e07389b797c791b08761183bcb479bd6b383662dd435a"
May 25 11:40:39.857122 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:40:39.856635    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd56801d6efc44f41a3e07389b797c791b08761183bcb479bd6b383662dd435a\": container with ID starting with cd56801d6efc44f41a3e07389b797c791b08761183bcb479bd6b383662dd435a not found: ID does not exist" containerID="cd56801d6efc44f41a3e07389b797c791b08761183bcb479bd6b383662dd435a"
May 25 11:40:39.857122 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:40:39.856665    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd56801d6efc44f41a3e07389b797c791b08761183bcb479bd6b383662dd435a"} err="failed to get container status \"cd56801d6efc44f41a3e07389b797c791b08761183bcb479bd6b383662dd435a\": rpc error: code = NotFound desc = could not find container \"cd56801d6efc44f41a3e07389b797c791b08761183bcb479bd6b383662dd435a\": container with ID starting with cd56801d6efc44f41a3e07389b797c791b08761183bcb479bd6b383662dd435a not found: ID does not exist"
May 25 11:40:39.857122 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:40:39.856689    2576 scope.go:117] "RemoveContainer" containerID="d2dfed07f97d309cb9295959a696a162ecbf0c37c84deb20a02c172733ed5fc5"
May 25 11:40:39.859179 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:40:39.859139    2576 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-66b9664c88slmlh"]
May 25 11:40:39.862143 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:40:39.862118    2576 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-66b9664c88slmlh"]
May 25 11:40:39.866950 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:40:39.866932    2576 scope.go:117] "RemoveContainer" containerID="01220baf28b056dfe7bc5f90266fc51d887398914ac3ccf6dcce2e09c250053f"
May 25 11:40:39.872478 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:40:39.872455    2576 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-cdng2p5"]
May 25 11:40:39.877482 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:40:39.877457    2576 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-cdng2p5"]
May 25 11:40:39.934724 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:40:39.934698    2576 scope.go:117] "RemoveContainer" containerID="ec8382e356e0a5f26fb5ad1ede5104af1e35c44b9c7116f1bf0b54f7079f7b23"
May 25 11:40:39.944030 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:40:39.944010    2576 scope.go:117] "RemoveContainer" containerID="d2dfed07f97d309cb9295959a696a162ecbf0c37c84deb20a02c172733ed5fc5"
May 25 11:40:39.944290 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:40:39.944260    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2dfed07f97d309cb9295959a696a162ecbf0c37c84deb20a02c172733ed5fc5\": container with ID starting with d2dfed07f97d309cb9295959a696a162ecbf0c37c84deb20a02c172733ed5fc5 not found: ID does not exist" containerID="d2dfed07f97d309cb9295959a696a162ecbf0c37c84deb20a02c172733ed5fc5"
May 25 11:40:39.944352 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:40:39.944308    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2dfed07f97d309cb9295959a696a162ecbf0c37c84deb20a02c172733ed5fc5"} err="failed to get container status \"d2dfed07f97d309cb9295959a696a162ecbf0c37c84deb20a02c172733ed5fc5\": rpc error: code = NotFound desc = could not find container \"d2dfed07f97d309cb9295959a696a162ecbf0c37c84deb20a02c172733ed5fc5\": container with ID starting with d2dfed07f97d309cb9295959a696a162ecbf0c37c84deb20a02c172733ed5fc5 not found: ID does not exist"
May 25 11:40:39.944352 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:40:39.944330    2576 scope.go:117] "RemoveContainer" containerID="01220baf28b056dfe7bc5f90266fc51d887398914ac3ccf6dcce2e09c250053f"
May 25 11:40:39.944613 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:40:39.944592    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01220baf28b056dfe7bc5f90266fc51d887398914ac3ccf6dcce2e09c250053f\": container with ID starting with 01220baf28b056dfe7bc5f90266fc51d887398914ac3ccf6dcce2e09c250053f not found: ID does not exist" containerID="01220baf28b056dfe7bc5f90266fc51d887398914ac3ccf6dcce2e09c250053f"
May 25 11:40:39.944689 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:40:39.944624    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01220baf28b056dfe7bc5f90266fc51d887398914ac3ccf6dcce2e09c250053f"} err="failed to get container status \"01220baf28b056dfe7bc5f90266fc51d887398914ac3ccf6dcce2e09c250053f\": rpc error: code = NotFound desc = could not find container \"01220baf28b056dfe7bc5f90266fc51d887398914ac3ccf6dcce2e09c250053f\": container with ID starting with 01220baf28b056dfe7bc5f90266fc51d887398914ac3ccf6dcce2e09c250053f not found: ID does not exist"
May 25 11:40:39.944689 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:40:39.944652    2576 scope.go:117] "RemoveContainer" containerID="ec8382e356e0a5f26fb5ad1ede5104af1e35c44b9c7116f1bf0b54f7079f7b23"
May 25 11:40:39.944922 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:40:39.944901    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec8382e356e0a5f26fb5ad1ede5104af1e35c44b9c7116f1bf0b54f7079f7b23\": container with ID starting with ec8382e356e0a5f26fb5ad1ede5104af1e35c44b9c7116f1bf0b54f7079f7b23 not found: ID does not exist" containerID="ec8382e356e0a5f26fb5ad1ede5104af1e35c44b9c7116f1bf0b54f7079f7b23"
May 25 11:40:39.945039 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:40:39.944923    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec8382e356e0a5f26fb5ad1ede5104af1e35c44b9c7116f1bf0b54f7079f7b23"} err="failed to get container status \"ec8382e356e0a5f26fb5ad1ede5104af1e35c44b9c7116f1bf0b54f7079f7b23\": rpc error: code = NotFound desc = could not find container \"ec8382e356e0a5f26fb5ad1ede5104af1e35c44b9c7116f1bf0b54f7079f7b23\": container with ID starting with ec8382e356e0a5f26fb5ad1ede5104af1e35c44b9c7116f1bf0b54f7079f7b23 not found: ID does not exist"
May 25 11:40:39.945039 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:40:39.944938    2576 scope.go:117] "RemoveContainer" containerID="d2dfed07f97d309cb9295959a696a162ecbf0c37c84deb20a02c172733ed5fc5"
May 25 11:40:39.945217 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:40:39.945197    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2dfed07f97d309cb9295959a696a162ecbf0c37c84deb20a02c172733ed5fc5"} err="failed to get container status \"d2dfed07f97d309cb9295959a696a162ecbf0c37c84deb20a02c172733ed5fc5\": rpc error: code = NotFound desc = could not find container \"d2dfed07f97d309cb9295959a696a162ecbf0c37c84deb20a02c172733ed5fc5\": container with ID starting with d2dfed07f97d309cb9295959a696a162ecbf0c37c84deb20a02c172733ed5fc5 not found: ID does not exist"
May 25 11:40:39.945217 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:40:39.945215    2576 scope.go:117] "RemoveContainer" containerID="01220baf28b056dfe7bc5f90266fc51d887398914ac3ccf6dcce2e09c250053f"
May 25 11:40:39.945449 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:40:39.945428    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01220baf28b056dfe7bc5f90266fc51d887398914ac3ccf6dcce2e09c250053f"} err="failed to get container status \"01220baf28b056dfe7bc5f90266fc51d887398914ac3ccf6dcce2e09c250053f\": rpc error: code = NotFound desc = could not find container \"01220baf28b056dfe7bc5f90266fc51d887398914ac3ccf6dcce2e09c250053f\": container with ID starting with 01220baf28b056dfe7bc5f90266fc51d887398914ac3ccf6dcce2e09c250053f not found: ID does not exist"
May 25 11:40:39.945449 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:40:39.945447    2576 scope.go:117] "RemoveContainer" containerID="ec8382e356e0a5f26fb5ad1ede5104af1e35c44b9c7116f1bf0b54f7079f7b23"
May 25 11:40:39.945713 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:40:39.945682    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec8382e356e0a5f26fb5ad1ede5104af1e35c44b9c7116f1bf0b54f7079f7b23"} err="failed to get container status \"ec8382e356e0a5f26fb5ad1ede5104af1e35c44b9c7116f1bf0b54f7079f7b23\": rpc error: code = NotFound desc = could not find container \"ec8382e356e0a5f26fb5ad1ede5104af1e35c44b9c7116f1bf0b54f7079f7b23\": container with ID starting with ec8382e356e0a5f26fb5ad1ede5104af1e35c44b9c7116f1bf0b54f7079f7b23 not found: ID does not exist"
May 25 11:40:41.099445 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:40:41.099414    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="599e3fe7-d07d-405b-820a-42e866f1636d" path="/var/lib/kubelet/pods/599e3fe7-d07d-405b-820a-42e866f1636d/volumes"
May 25 11:40:41.099860 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:40:41.099847    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a33d5fe5-3633-466f-888d-5250a697bed0" path="/var/lib/kubelet/pods/a33d5fe5-3633-466f-888d-5250a697bed0/volumes"
May 25 11:40:43.796011 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:40:43.795964    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-545854866-cq97p" podUID="77cb6ea6-0019-4f5a-a219-72854cf6c824" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/health\": dial tcp 10.133.0.36:8000: connect: connection refused"
May 25 11:40:43.800445 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:40:43.800387    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6b47c648bc-kgnng" podUID="1c0dfef3-38b4-4195-a07c-acab5c17e850" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8001/health\": dial tcp 10.133.0.37:8001: connect: connection refused"
May 25 11:40:53.796671 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:40:53.796618    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-545854866-cq97p" podUID="77cb6ea6-0019-4f5a-a219-72854cf6c824" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/health\": dial tcp 10.133.0.36:8000: connect: connection refused"
May 25 11:40:53.801675 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:40:53.801627    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6b47c648bc-kgnng" podUID="1c0dfef3-38b4-4195-a07c-acab5c17e850" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8001/health\": dial tcp 10.133.0.37:8001: connect: connection refused"
May 25 11:41:03.796241 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:41:03.796188    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-545854866-cq97p" podUID="77cb6ea6-0019-4f5a-a219-72854cf6c824" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/health\": dial tcp 10.133.0.36:8000: connect: connection refused"
May 25 11:41:03.800637 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:41:03.800593    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6b47c648bc-kgnng" podUID="1c0dfef3-38b4-4195-a07c-acab5c17e850" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8001/health\": dial tcp 10.133.0.37:8001: connect: connection refused"
May 25 11:41:13.796459 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:41:13.796406    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-545854866-cq97p" podUID="77cb6ea6-0019-4f5a-a219-72854cf6c824" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/health\": dial tcp 10.133.0.36:8000: connect: connection refused"
May 25 11:41:13.800928 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:41:13.800860    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6b47c648bc-kgnng" podUID="1c0dfef3-38b4-4195-a07c-acab5c17e850" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8001/health\": dial tcp 10.133.0.37:8001: connect: connection refused"
May 25 11:41:23.796363 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:41:23.796263    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-545854866-cq97p" podUID="77cb6ea6-0019-4f5a-a219-72854cf6c824" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/health\": dial tcp 10.133.0.36:8000: connect: connection refused"
May 25 11:41:23.800744 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:41:23.800707    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6b47c648bc-kgnng" podUID="1c0dfef3-38b4-4195-a07c-acab5c17e850" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8001/health\": dial tcp 10.133.0.37:8001: connect: connection refused"
May 25 11:41:33.795901 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:41:33.795848    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-545854866-cq97p" podUID="77cb6ea6-0019-4f5a-a219-72854cf6c824" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/health\": dial tcp 10.133.0.36:8000: connect: connection refused"
May 25 11:41:33.800117 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:41:33.800091    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6b47c648bc-kgnng" podUID="1c0dfef3-38b4-4195-a07c-acab5c17e850" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8001/health\": dial tcp 10.133.0.37:8001: connect: connection refused"
May 25 11:41:43.796201 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:41:43.796159    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-545854866-cq97p" podUID="77cb6ea6-0019-4f5a-a219-72854cf6c824" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/health\": dial tcp 10.133.0.36:8000: connect: connection refused"
May 25 11:41:43.800632 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:41:43.800585    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6b47c648bc-kgnng" podUID="1c0dfef3-38b4-4195-a07c-acab5c17e850" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8001/health\": dial tcp 10.133.0.37:8001: connect: connection refused"
May 25 11:41:53.796492 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:41:53.796436    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-545854866-cq97p" podUID="77cb6ea6-0019-4f5a-a219-72854cf6c824" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/health\": dial tcp 10.133.0.36:8000: connect: connection refused"
May 25 11:41:53.800837 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:41:53.800790    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6b47c648bc-kgnng" podUID="1c0dfef3-38b4-4195-a07c-acab5c17e850" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8001/health\": dial tcp 10.133.0.37:8001: connect: connection refused"
May 25 11:42:03.796256 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:42:03.796214    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-545854866-cq97p" podUID="77cb6ea6-0019-4f5a-a219-72854cf6c824" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/health\": dial tcp 10.133.0.36:8000: connect: connection refused"
May 25 11:42:03.800668 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:42:03.800632    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6b47c648bc-kgnng" podUID="1c0dfef3-38b4-4195-a07c-acab5c17e850" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8001/health\": dial tcp 10.133.0.37:8001: connect: connection refused"
May 25 11:42:13.796084 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:42:13.796040    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-545854866-cq97p" podUID="77cb6ea6-0019-4f5a-a219-72854cf6c824" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/health\": dial tcp 10.133.0.36:8000: connect: connection refused"
May 25 11:42:13.800453 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:42:13.800419    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6b47c648bc-kgnng" podUID="1c0dfef3-38b4-4195-a07c-acab5c17e850" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8001/health\": dial tcp 10.133.0.37:8001: connect: connection refused"
May 25 11:42:23.796574 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:42:23.796530    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-545854866-cq97p" podUID="77cb6ea6-0019-4f5a-a219-72854cf6c824" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/health\": dial tcp 10.133.0.36:8000: connect: connection refused"
May 25 11:42:23.801057 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:42:23.801019    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6b47c648bc-kgnng" podUID="1c0dfef3-38b4-4195-a07c-acab5c17e850" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8001/health\": dial tcp 10.133.0.37:8001: connect: connection refused"
May 25 11:42:33.796643 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:42:33.796601    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-545854866-cq97p" podUID="77cb6ea6-0019-4f5a-a219-72854cf6c824" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/health\": dial tcp 10.133.0.36:8000: connect: connection refused"
May 25 11:42:33.800969 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:42:33.800920    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6b47c648bc-kgnng" podUID="1c0dfef3-38b4-4195-a07c-acab5c17e850" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8001/health\": dial tcp 10.133.0.37:8001: connect: connection refused"
May 25 11:42:35.116387 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:42:35.116358    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h6b5t_12b45353-8444-41fc-a8ed-fb691764eb63/ovn-acl-logging/0.log"
May 25 11:42:35.117915 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:42:35.117892    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h6b5t_12b45353-8444-41fc-a8ed-fb691764eb63/ovn-acl-logging/0.log"
May 25 11:42:43.806771 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:42:43.806740    2576 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-545854866-cq97p"
May 25 11:42:43.811140 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:42:43.811117    2576 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6b47c648bc-kgnng"
May 25 11:42:43.815610 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:42:43.815588    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-545854866-cq97p"
May 25 11:42:43.830503 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:42:43.830483    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6b47c648bc-kgnng"
May 25 11:42:51.554030 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:42:51.553994    2576 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-545854866-cq97p"]
May 25 11:42:51.554609 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:42:51.554344    2576 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-545854866-cq97p" podUID="77cb6ea6-0019-4f5a-a219-72854cf6c824" containerName="main" containerID="cri-o://838bbab20ca93f7029560b5dfadda21e257c2acd4c1b1deb3d37563855a85dfd" gracePeriod=30
May 25 11:42:51.563759 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:42:51.563731    2576 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6b47c648bc-kgnng"]
May 25 11:42:51.564186 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:42:51.564130    2576 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6b47c648bc-kgnng" podUID="1c0dfef3-38b4-4195-a07c-acab5c17e850" containerName="main" containerID="cri-o://4f8533625cf9d1f67bd9e991c79e026513dd35dbc43158603a589b1f2dc54fc9" gracePeriod=30
May 25 11:43:01.182064 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:43:01.182030    2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-799dc8b8c-8w6n5"]
May 25 11:43:01.182651 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:43:01.182420    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a33d5fe5-3633-466f-888d-5250a697bed0" containerName="llm-d-routing-sidecar"
May 25 11:43:01.182651 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:43:01.182434    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="a33d5fe5-3633-466f-888d-5250a697bed0" containerName="llm-d-routing-sidecar"
May 25 11:43:01.182651 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:43:01.182449    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a33d5fe5-3633-466f-888d-5250a697bed0" containerName="main"
May 25 11:43:01.182651 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:43:01.182459    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="a33d5fe5-3633-466f-888d-5250a697bed0" containerName="main"
May 25 11:43:01.182651 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:43:01.182484    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="599e3fe7-d07d-405b-820a-42e866f1636d" containerName="storage-initializer"
May 25 11:43:01.182651 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:43:01.182493    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="599e3fe7-d07d-405b-820a-42e866f1636d" containerName="storage-initializer"
May 25 11:43:01.182651 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:43:01.182508    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a33d5fe5-3633-466f-888d-5250a697bed0" containerName="storage-initializer"
May 25 11:43:01.182651 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:43:01.182514    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="a33d5fe5-3633-466f-888d-5250a697bed0" containerName="storage-initializer"
May 25 11:43:01.182651 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:43:01.182529    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="599e3fe7-d07d-405b-820a-42e866f1636d" containerName="main"
May 25 11:43:01.182651 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:43:01.182537    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="599e3fe7-d07d-405b-820a-42e866f1636d" containerName="main"
May 25 11:43:01.182651 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:43:01.182597    2576 memory_manager.go:356] "RemoveStaleState removing state" podUID="a33d5fe5-3633-466f-888d-5250a697bed0" containerName="llm-d-routing-sidecar"
May 25 11:43:01.182651 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:43:01.182613    2576 memory_manager.go:356] "RemoveStaleState removing state" podUID="599e3fe7-d07d-405b-820a-42e866f1636d" containerName="main"
May 25 11:43:01.182651 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:43:01.182622    2576 memory_manager.go:356] "RemoveStaleState removing state" podUID="a33d5fe5-3633-466f-888d-5250a697bed0" containerName="main"
May 25 11:43:01.184608 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:43:01.184592    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-799dc8b8c-8w6n5"
May 25 11:43:01.187542 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:43:01.187520    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"router-with-refs-pd-test-kserve-dockercfg-bkfdj\""
May 25 11:43:01.187652 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:43:01.187567    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"router-with-refs-pd-test-kserve-self-signed-certs\""
May 25 11:43:01.197414 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:43:01.197388    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e96af524-82a6-41ef-8bf2-8a1bf59bc33f-model-cache\") pod \"router-with-refs-pd-test-kserve-799dc8b8c-8w6n5\" (UID: \"e96af524-82a6-41ef-8bf2-8a1bf59bc33f\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-799dc8b8c-8w6n5"
May 25 11:43:01.197537 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:43:01.197482    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e96af524-82a6-41ef-8bf2-8a1bf59bc33f-tmp-dir\") pod \"router-with-refs-pd-test-kserve-799dc8b8c-8w6n5\" (UID: \"e96af524-82a6-41ef-8bf2-8a1bf59bc33f\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-799dc8b8c-8w6n5"
May 25 11:43:01.197537 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:43:01.197511    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e96af524-82a6-41ef-8bf2-8a1bf59bc33f-tls-certs\") pod \"router-with-refs-pd-test-kserve-799dc8b8c-8w6n5\" (UID: \"e96af524-82a6-41ef-8bf2-8a1bf59bc33f\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-799dc8b8c-8w6n5"
May 25 11:43:01.197637 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:43:01.197547    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e96af524-82a6-41ef-8bf2-8a1bf59bc33f-home\") pod \"router-with-refs-pd-test-kserve-799dc8b8c-8w6n5\" (UID: \"e96af524-82a6-41ef-8bf2-8a1bf59bc33f\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-799dc8b8c-8w6n5"
May 25 11:43:01.197637 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:43:01.197620    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e96af524-82a6-41ef-8bf2-8a1bf59bc33f-dshm\") pod \"router-with-refs-pd-test-kserve-799dc8b8c-8w6n5\" (UID: \"e96af524-82a6-41ef-8bf2-8a1bf59bc33f\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-799dc8b8c-8w6n5"
May 25 11:43:01.197721 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:43:01.197651    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e96af524-82a6-41ef-8bf2-8a1bf59bc33f-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-799dc8b8c-8w6n5\" (UID: \"e96af524-82a6-41ef-8bf2-8a1bf59bc33f\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-799dc8b8c-8w6n5"
May 25 11:43:01.197721 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:43:01.197694    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h824v\" (UniqueName: \"kubernetes.io/projected/e96af524-82a6-41ef-8bf2-8a1bf59bc33f-kube-api-access-h824v\") pod \"router-with-refs-pd-test-kserve-799dc8b8c-8w6n5\" (UID: \"e96af524-82a6-41ef-8bf2-8a1bf59bc33f\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-799dc8b8c-8w6n5"
May 25 11:43:01.203587 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:43:01.203565    2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-78459cb79d-92lfb"]
May 25 11:43:01.206085 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:43:01.206061    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-78459cb79d-92lfb"
May 25 11:43:01.210427 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:43:01.210393    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-799dc8b8c-8w6n5"]
May 25 11:43:01.225622 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:43:01.225593    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-78459cb79d-92lfb"]
May 25 11:43:01.298709 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:43:01.298670    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/887d4410-aee9-44bc-99ee-9fa600b7b6ac-dshm\") pod \"router-with-refs-pd-test-kserve-prefill-78459cb79d-92lfb\" (UID: \"887d4410-aee9-44bc-99ee-9fa600b7b6ac\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-78459cb79d-92lfb"
May 25 11:43:01.298709 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:43:01.298714    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e96af524-82a6-41ef-8bf2-8a1bf59bc33f-dshm\") pod \"router-with-refs-pd-test-kserve-799dc8b8c-8w6n5\" (UID: \"e96af524-82a6-41ef-8bf2-8a1bf59bc33f\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-799dc8b8c-8w6n5"
May 25 11:43:01.298933 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:43:01.298736    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e96af524-82a6-41ef-8bf2-8a1bf59bc33f-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-799dc8b8c-8w6n5\" (UID: \"e96af524-82a6-41ef-8bf2-8a1bf59bc33f\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-799dc8b8c-8w6n5"
May 25 11:43:01.298933 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:43:01.298802    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6qmg\" (UniqueName: \"kubernetes.io/projected/887d4410-aee9-44bc-99ee-9fa600b7b6ac-kube-api-access-w6qmg\") pod \"router-with-refs-pd-test-kserve-prefill-78459cb79d-92lfb\" (UID: \"887d4410-aee9-44bc-99ee-9fa600b7b6ac\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-78459cb79d-92lfb"
May 25 11:43:01.298933 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:43:01.298838    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/887d4410-aee9-44bc-99ee-9fa600b7b6ac-tmp-dir\") pod \"router-with-refs-pd-test-kserve-prefill-78459cb79d-92lfb\" (UID: \"887d4410-aee9-44bc-99ee-9fa600b7b6ac\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-78459cb79d-92lfb"
May 25 11:43:01.298933 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:43:01.298902    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e96af524-82a6-41ef-8bf2-8a1bf59bc33f-home\") pod \"router-with-refs-pd-test-kserve-799dc8b8c-8w6n5\" (UID: \"e96af524-82a6-41ef-8bf2-8a1bf59bc33f\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-799dc8b8c-8w6n5"
May 25 11:43:01.299076 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:43:01.298933    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-h824v\" (UniqueName: \"kubernetes.io/projected/e96af524-82a6-41ef-8bf2-8a1bf59bc33f-kube-api-access-h824v\") pod \"router-with-refs-pd-test-kserve-799dc8b8c-8w6n5\" (UID: \"e96af524-82a6-41ef-8bf2-8a1bf59bc33f\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-799dc8b8c-8w6n5"
May 25 11:43:01.299076 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:43:01.298976    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/887d4410-aee9-44bc-99ee-9fa600b7b6ac-tls-certs\") pod \"router-with-refs-pd-test-kserve-prefill-78459cb79d-92lfb\" (UID: \"887d4410-aee9-44bc-99ee-9fa600b7b6ac\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-78459cb79d-92lfb"
May 25 11:43:01.299076 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:43:01.299008    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/887d4410-aee9-44bc-99ee-9fa600b7b6ac-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-prefill-78459cb79d-92lfb\" (UID: \"887d4410-aee9-44bc-99ee-9fa600b7b6ac\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-78459cb79d-92lfb"
May 25 11:43:01.299327 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:43:01.299082    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e96af524-82a6-41ef-8bf2-8a1bf59bc33f-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-799dc8b8c-8w6n5\" (UID: \"e96af524-82a6-41ef-8bf2-8a1bf59bc33f\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-799dc8b8c-8w6n5"
May 25 11:43:01.299327 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:43:01.299175    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e96af524-82a6-41ef-8bf2-8a1bf59bc33f-model-cache\") pod \"router-with-refs-pd-test-kserve-799dc8b8c-8w6n5\" (UID: \"e96af524-82a6-41ef-8bf2-8a1bf59bc33f\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-799dc8b8c-8w6n5"
May 25 11:43:01.299327 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:43:01.299190    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e96af524-82a6-41ef-8bf2-8a1bf59bc33f-home\") pod \"router-with-refs-pd-test-kserve-799dc8b8c-8w6n5\" (UID: \"e96af524-82a6-41ef-8bf2-8a1bf59bc33f\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-799dc8b8c-8w6n5"
May 25 11:43:01.299327 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:43:01.299229    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/887d4410-aee9-44bc-99ee-9fa600b7b6ac-home\") pod \"router-with-refs-pd-test-kserve-prefill-78459cb79d-92lfb\" (UID: \"887d4410-aee9-44bc-99ee-9fa600b7b6ac\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-78459cb79d-92lfb"
May 25 11:43:01.299327 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:43:01.299316    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/887d4410-aee9-44bc-99ee-9fa600b7b6ac-model-cache\") pod \"router-with-refs-pd-test-kserve-prefill-78459cb79d-92lfb\" (UID: \"887d4410-aee9-44bc-99ee-9fa600b7b6ac\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-78459cb79d-92lfb"
May 25 11:43:01.299517 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:43:01.299375    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e96af524-82a6-41ef-8bf2-8a1bf59bc33f-model-cache\") pod \"router-with-refs-pd-test-kserve-799dc8b8c-8w6n5\" (UID: \"e96af524-82a6-41ef-8bf2-8a1bf59bc33f\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-799dc8b8c-8w6n5"
May 25 11:43:01.299517 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:43:01.299375    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e96af524-82a6-41ef-8bf2-8a1bf59bc33f-tmp-dir\") pod \"router-with-refs-pd-test-kserve-799dc8b8c-8w6n5\" (UID: \"e96af524-82a6-41ef-8bf2-8a1bf59bc33f\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-799dc8b8c-8w6n5"
May 25 11:43:01.299517 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:43:01.299431    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e96af524-82a6-41ef-8bf2-8a1bf59bc33f-tls-certs\") pod \"router-with-refs-pd-test-kserve-799dc8b8c-8w6n5\" (UID: \"e96af524-82a6-41ef-8bf2-8a1bf59bc33f\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-799dc8b8c-8w6n5"
May 25 11:43:01.299654 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:43:01.299635    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e96af524-82a6-41ef-8bf2-8a1bf59bc33f-tmp-dir\") pod \"router-with-refs-pd-test-kserve-799dc8b8c-8w6n5\" (UID: \"e96af524-82a6-41ef-8bf2-8a1bf59bc33f\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-799dc8b8c-8w6n5"
May 25 11:43:01.301094 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:43:01.301072    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e96af524-82a6-41ef-8bf2-8a1bf59bc33f-dshm\") pod \"router-with-refs-pd-test-kserve-799dc8b8c-8w6n5\" (UID: \"e96af524-82a6-41ef-8bf2-8a1bf59bc33f\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-799dc8b8c-8w6n5"
May 25 11:43:01.301843 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:43:01.301821    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e96af524-82a6-41ef-8bf2-8a1bf59bc33f-tls-certs\") pod \"router-with-refs-pd-test-kserve-799dc8b8c-8w6n5\" (UID: \"e96af524-82a6-41ef-8bf2-8a1bf59bc33f\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-799dc8b8c-8w6n5"
May 25 11:43:01.310772 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:43:01.310748    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h824v\" (UniqueName: \"kubernetes.io/projected/e96af524-82a6-41ef-8bf2-8a1bf59bc33f-kube-api-access-h824v\") pod \"router-with-refs-pd-test-kserve-799dc8b8c-8w6n5\" (UID: \"e96af524-82a6-41ef-8bf2-8a1bf59bc33f\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-799dc8b8c-8w6n5"
May 25 11:43:01.400304 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:43:01.400260    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/887d4410-aee9-44bc-99ee-9fa600b7b6ac-tls-certs\") pod \"router-with-refs-pd-test-kserve-prefill-78459cb79d-92lfb\" (UID: \"887d4410-aee9-44bc-99ee-9fa600b7b6ac\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-78459cb79d-92lfb"
May 25 11:43:01.400304 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:43:01.400305    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/887d4410-aee9-44bc-99ee-9fa600b7b6ac-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-prefill-78459cb79d-92lfb\" (UID: \"887d4410-aee9-44bc-99ee-9fa600b7b6ac\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-78459cb79d-92lfb"
May 25 11:43:01.400562 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:43:01.400327    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/887d4410-aee9-44bc-99ee-9fa600b7b6ac-home\") pod \"router-with-refs-pd-test-kserve-prefill-78459cb79d-92lfb\" (UID: \"887d4410-aee9-44bc-99ee-9fa600b7b6ac\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-78459cb79d-92lfb"
May 25 11:43:01.400562 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:43:01.400346    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/887d4410-aee9-44bc-99ee-9fa600b7b6ac-model-cache\") pod \"router-with-refs-pd-test-kserve-prefill-78459cb79d-92lfb\" (UID: \"887d4410-aee9-44bc-99ee-9fa600b7b6ac\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-78459cb79d-92lfb"
May 25 11:43:01.400562 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:43:01.400470    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/887d4410-aee9-44bc-99ee-9fa600b7b6ac-dshm\") pod \"router-with-refs-pd-test-kserve-prefill-78459cb79d-92lfb\" (UID: \"887d4410-aee9-44bc-99ee-9fa600b7b6ac\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-78459cb79d-92lfb"
May 25 11:43:01.400741 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:43:01.400709    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/887d4410-aee9-44bc-99ee-9fa600b7b6ac-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-prefill-78459cb79d-92lfb\" (UID: \"887d4410-aee9-44bc-99ee-9fa600b7b6ac\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-78459cb79d-92lfb"
May 25 11:43:01.400801 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:43:01.400754    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/887d4410-aee9-44bc-99ee-9fa600b7b6ac-home\") pod \"router-with-refs-pd-test-kserve-prefill-78459cb79d-92lfb\" (UID: \"887d4410-aee9-44bc-99ee-9fa600b7b6ac\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-78459cb79d-92lfb"
May 25 11:43:01.400801 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:43:01.400772    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-w6qmg\" (UniqueName: \"kubernetes.io/projected/887d4410-aee9-44bc-99ee-9fa600b7b6ac-kube-api-access-w6qmg\") pod \"router-with-refs-pd-test-kserve-prefill-78459cb79d-92lfb\" (UID: \"887d4410-aee9-44bc-99ee-9fa600b7b6ac\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-78459cb79d-92lfb"
May 25 11:43:01.400903 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:43:01.400808    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/887d4410-aee9-44bc-99ee-9fa600b7b6ac-model-cache\") pod \"router-with-refs-pd-test-kserve-prefill-78459cb79d-92lfb\" (UID: \"887d4410-aee9-44bc-99ee-9fa600b7b6ac\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-78459cb79d-92lfb"
May 25 11:43:01.400903 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:43:01.400818    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/887d4410-aee9-44bc-99ee-9fa600b7b6ac-tmp-dir\") pod \"router-with-refs-pd-test-kserve-prefill-78459cb79d-92lfb\" (UID: \"887d4410-aee9-44bc-99ee-9fa600b7b6ac\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-78459cb79d-92lfb"
May 25 11:43:01.401134 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:43:01.401114    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/887d4410-aee9-44bc-99ee-9fa600b7b6ac-tmp-dir\") pod \"router-with-refs-pd-test-kserve-prefill-78459cb79d-92lfb\" (UID: \"887d4410-aee9-44bc-99ee-9fa600b7b6ac\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-78459cb79d-92lfb"
May 25 11:43:01.402659 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:43:01.402641    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/887d4410-aee9-44bc-99ee-9fa600b7b6ac-dshm\") pod \"router-with-refs-pd-test-kserve-prefill-78459cb79d-92lfb\" (UID: \"887d4410-aee9-44bc-99ee-9fa600b7b6ac\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-78459cb79d-92lfb"
May 25 11:43:01.402958 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:43:01.402938    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/887d4410-aee9-44bc-99ee-9fa600b7b6ac-tls-certs\") pod \"router-with-refs-pd-test-kserve-prefill-78459cb79d-92lfb\" (UID: \"887d4410-aee9-44bc-99ee-9fa600b7b6ac\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-78459cb79d-92lfb"
May 25 11:43:01.410221 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:43:01.410197    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6qmg\" (UniqueName: \"kubernetes.io/projected/887d4410-aee9-44bc-99ee-9fa600b7b6ac-kube-api-access-w6qmg\") pod \"router-with-refs-pd-test-kserve-prefill-78459cb79d-92lfb\" (UID: \"887d4410-aee9-44bc-99ee-9fa600b7b6ac\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-78459cb79d-92lfb"
May 25 11:43:01.494641 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:43:01.494556    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-799dc8b8c-8w6n5"
May 25 11:43:01.519169 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:43:01.519110    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-78459cb79d-92lfb"
May 25 11:43:01.649449 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:43:01.649412    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-799dc8b8c-8w6n5"]
May 25 11:43:01.652884 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:43:01.652854    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode96af524_82a6_41ef_8bf2_8a1bf59bc33f.slice/crio-aae05d56c6e7bfe3d01ab60827ea6f64dd9bff60c5918d6eab8862266a053e48 WatchSource:0}: Error finding container aae05d56c6e7bfe3d01ab60827ea6f64dd9bff60c5918d6eab8862266a053e48: Status 404 returned error can't find the container with id aae05d56c6e7bfe3d01ab60827ea6f64dd9bff60c5918d6eab8862266a053e48
May 25 11:43:01.654806 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:43:01.654788    2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 25 11:43:01.675594 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:43:01.675562    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-78459cb79d-92lfb"]
May 25 11:43:01.683557 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:43:01.683523    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod887d4410_aee9_44bc_99ee_9fa600b7b6ac.slice/crio-a47be0c9e000bc6f5b13ae325bda2c7bcade6c8322e8aa43f6b1fe42558b768d WatchSource:0}: Error finding container a47be0c9e000bc6f5b13ae325bda2c7bcade6c8322e8aa43f6b1fe42558b768d: Status 404 returned error can't find the container with id a47be0c9e000bc6f5b13ae325bda2c7bcade6c8322e8aa43f6b1fe42558b768d
May 25 11:43:02.349267 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:43:02.341243    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-799dc8b8c-8w6n5" event={"ID":"e96af524-82a6-41ef-8bf2-8a1bf59bc33f","Type":"ContainerStarted","Data":"ccd7f1468d92f06063bfdd6d854e68e1b9b7d93b085f4b90887f8eddc795a286"}
May 25 11:43:02.349267 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:43:02.341738    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-799dc8b8c-8w6n5" event={"ID":"e96af524-82a6-41ef-8bf2-8a1bf59bc33f","Type":"ContainerStarted","Data":"aae05d56c6e7bfe3d01ab60827ea6f64dd9bff60c5918d6eab8862266a053e48"}
May 25 11:43:02.349267 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:43:02.342309    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-799dc8b8c-8w6n5"
May 25 11:43:02.351954 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:43:02.351925    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-78459cb79d-92lfb" event={"ID":"887d4410-aee9-44bc-99ee-9fa600b7b6ac","Type":"ContainerStarted","Data":"2d12a14fe595187f73cf5b17df0af7f59862b696ca2b1d9e5a5fb797cbc9eb93"}
May 25 11:43:02.352098 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:43:02.351965    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-78459cb79d-92lfb" event={"ID":"887d4410-aee9-44bc-99ee-9fa600b7b6ac","Type":"ContainerStarted","Data":"a47be0c9e000bc6f5b13ae325bda2c7bcade6c8322e8aa43f6b1fe42558b768d"}
May 25 11:43:03.359501 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:43:03.359453    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-799dc8b8c-8w6n5" event={"ID":"e96af524-82a6-41ef-8bf2-8a1bf59bc33f","Type":"ContainerStarted","Data":"e1a0f315b50db54160ac4a8e7ce07441060df2a3c37274f4ecfbee9a15763540"}
May 25 11:43:08.382267 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:43:08.382225    2576 generic.go:358] "Generic (PLEG): container finished" podID="887d4410-aee9-44bc-99ee-9fa600b7b6ac" containerID="2d12a14fe595187f73cf5b17df0af7f59862b696ca2b1d9e5a5fb797cbc9eb93" exitCode=0
May 25 11:43:08.382710 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:43:08.382288    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-78459cb79d-92lfb" event={"ID":"887d4410-aee9-44bc-99ee-9fa600b7b6ac","Type":"ContainerDied","Data":"2d12a14fe595187f73cf5b17df0af7f59862b696ca2b1d9e5a5fb797cbc9eb93"}
May 25 11:43:09.387225 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:43:09.387190    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-78459cb79d-92lfb" event={"ID":"887d4410-aee9-44bc-99ee-9fa600b7b6ac","Type":"ContainerStarted","Data":"bd4f18fb0669d2bc012182ce6c58348fb7fdd977ce0815d73a782ca0e48bc728"}
May 25 11:43:09.414649 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:43:09.414592    2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-78459cb79d-92lfb" podStartSLOduration=8.414573792 podStartE2EDuration="8.414573792s" podCreationTimestamp="2026-05-25 11:43:01 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 11:43:09.411165889 +0000 UTC m=+1534.928897346" watchObservedRunningTime="2026-05-25 11:43:09.414573792 +0000 UTC m=+1534.932305249"
May 25 11:43:11.519849 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:43:11.519802    2576 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-78459cb79d-92lfb"
May 25 11:43:11.519849 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:43:11.519846    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-78459cb79d-92lfb"
May 25 11:43:11.521163 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:43:11.521131    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-78459cb79d-92lfb" podUID="887d4410-aee9-44bc-99ee-9fa600b7b6ac" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/health\": dial tcp 10.133.0.39:8000: connect: connection refused"
May 25 11:43:14.375891 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:43:14.375861    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-799dc8b8c-8w6n5"
May 25 11:43:19.426591 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:43:19.426563    2576 generic.go:358] "Generic (PLEG): container finished" podID="e96af524-82a6-41ef-8bf2-8a1bf59bc33f" containerID="e1a0f315b50db54160ac4a8e7ce07441060df2a3c37274f4ecfbee9a15763540" exitCode=0
May 25 11:43:19.426963 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:43:19.426638    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-799dc8b8c-8w6n5" event={"ID":"e96af524-82a6-41ef-8bf2-8a1bf59bc33f","Type":"ContainerDied","Data":"e1a0f315b50db54160ac4a8e7ce07441060df2a3c37274f4ecfbee9a15763540"}
May 25 11:43:20.439723 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:43:20.439682    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-799dc8b8c-8w6n5" event={"ID":"e96af524-82a6-41ef-8bf2-8a1bf59bc33f","Type":"ContainerStarted","Data":"b60aa49e873e79bbea2ce5a73b8123ed1f3fcac0abc08b5393ed0aaf61aa767d"}
May 25 11:43:20.468167 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:43:20.468108    2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-799dc8b8c-8w6n5" podStartSLOduration=19.468089881 podStartE2EDuration="19.468089881s" podCreationTimestamp="2026-05-25 11:43:01 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 11:43:20.465001363 +0000 UTC m=+1545.982732818" watchObservedRunningTime="2026-05-25 11:43:20.468089881 +0000 UTC m=+1545.985821338"
May 25 11:43:21.494801 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:43:21.494761    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-799dc8b8c-8w6n5"
May 25 11:43:21.495230 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:43:21.494813    2576 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-799dc8b8c-8w6n5"
May 25 11:43:21.495230 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:43:21.495025    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-799dc8b8c-8w6n5" podUID="e96af524-82a6-41ef-8bf2-8a1bf59bc33f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8001/health\": dial tcp 10.133.0.38:8001: connect: connection refused"
May 25 11:43:21.519685 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:43:21.519653    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-78459cb79d-92lfb" podUID="887d4410-aee9-44bc-99ee-9fa600b7b6ac" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/health\": dial tcp 10.133.0.39:8000: connect: connection refused"
May 25 11:43:21.564437 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:43:21.564396    2576 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6b47c648bc-kgnng" podUID="1c0dfef3-38b4-4195-a07c-acab5c17e850" containerName="llm-d-routing-sidecar" containerID="cri-o://6b34ee9d8e732ca40b6f96fa3f02559f62b2f50b0c96d9b4e7da362fcce8abe0" gracePeriod=2
May 25 11:43:21.927292 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:43:21.927251    2576 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-545854866-cq97p"
May 25 11:43:21.959065 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:43:21.959036    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_custom-route-timeout-pd-test-kserve-6b47c648bc-kgnng_1c0dfef3-38b4-4195-a07c-acab5c17e850/main/0.log"
May 25 11:43:21.959881 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:43:21.959862    2576 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6b47c648bc-kgnng"
May 25 11:43:22.013344 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:43:22.013306    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/77cb6ea6-0019-4f5a-a219-72854cf6c824-kserve-provision-location\") pod \"77cb6ea6-0019-4f5a-a219-72854cf6c824\" (UID: \"77cb6ea6-0019-4f5a-a219-72854cf6c824\") "
May 25 11:43:22.013516 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:43:22.013361    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/77cb6ea6-0019-4f5a-a219-72854cf6c824-home\") pod \"77cb6ea6-0019-4f5a-a219-72854cf6c824\" (UID: \"77cb6ea6-0019-4f5a-a219-72854cf6c824\") "
May 25 11:43:22.013516 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:43:22.013435    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/77cb6ea6-0019-4f5a-a219-72854cf6c824-tls-certs\") pod \"77cb6ea6-0019-4f5a-a219-72854cf6c824\" (UID: \"77cb6ea6-0019-4f5a-a219-72854cf6c824\") "
May 25 11:43:22.013516 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:43:22.013476    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/77cb6ea6-0019-4f5a-a219-72854cf6c824-model-cache\") pod \"77cb6ea6-0019-4f5a-a219-72854cf6c824\" (UID: \"77cb6ea6-0019-4f5a-a219-72854cf6c824\") "
May 25 11:43:22.013516 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:43:22.013502    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/77cb6ea6-0019-4f5a-a219-72854cf6c824-dshm\") pod \"77cb6ea6-0019-4f5a-a219-72854cf6c824\" (UID: \"77cb6ea6-0019-4f5a-a219-72854cf6c824\") "
May 25 11:43:22.013766 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:43:22.013528    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/77cb6ea6-0019-4f5a-a219-72854cf6c824-tmp-dir\") pod \"77cb6ea6-0019-4f5a-a219-72854cf6c824\" (UID: \"77cb6ea6-0019-4f5a-a219-72854cf6c824\") "
May 25 11:43:22.013766 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:43:22.013589    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tmgkb\" (UniqueName: \"kubernetes.io/projected/77cb6ea6-0019-4f5a-a219-72854cf6c824-kube-api-access-tmgkb\") pod \"77cb6ea6-0019-4f5a-a219-72854cf6c824\" (UID: \"77cb6ea6-0019-4f5a-a219-72854cf6c824\") "
May 25 11:43:22.014198 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:43:22.014159    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77cb6ea6-0019-4f5a-a219-72854cf6c824-home" (OuterVolumeSpecName: "home") pod "77cb6ea6-0019-4f5a-a219-72854cf6c824" (UID: "77cb6ea6-0019-4f5a-a219-72854cf6c824"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:43:22.014393 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:43:22.014363    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77cb6ea6-0019-4f5a-a219-72854cf6c824-model-cache" (OuterVolumeSpecName: "model-cache") pod "77cb6ea6-0019-4f5a-a219-72854cf6c824" (UID: "77cb6ea6-0019-4f5a-a219-72854cf6c824"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:43:22.017482 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:43:22.017411    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77cb6ea6-0019-4f5a-a219-72854cf6c824-kube-api-access-tmgkb" (OuterVolumeSpecName: "kube-api-access-tmgkb") pod "77cb6ea6-0019-4f5a-a219-72854cf6c824" (UID: "77cb6ea6-0019-4f5a-a219-72854cf6c824"). InnerVolumeSpecName "kube-api-access-tmgkb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 11:43:22.018482 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:43:22.018459    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/77cb6ea6-0019-4f5a-a219-72854cf6c824-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "77cb6ea6-0019-4f5a-a219-72854cf6c824" (UID: "77cb6ea6-0019-4f5a-a219-72854cf6c824"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 11:43:22.021392 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:43:22.021364    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77cb6ea6-0019-4f5a-a219-72854cf6c824-dshm" (OuterVolumeSpecName: "dshm") pod "77cb6ea6-0019-4f5a-a219-72854cf6c824" (UID: "77cb6ea6-0019-4f5a-a219-72854cf6c824"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:43:22.029685 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:43:22.029655    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77cb6ea6-0019-4f5a-a219-72854cf6c824-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "77cb6ea6-0019-4f5a-a219-72854cf6c824" (UID: "77cb6ea6-0019-4f5a-a219-72854cf6c824"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:43:22.079940 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:43:22.079877    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77cb6ea6-0019-4f5a-a219-72854cf6c824-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "77cb6ea6-0019-4f5a-a219-72854cf6c824" (UID: "77cb6ea6-0019-4f5a-a219-72854cf6c824"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:43:22.115099 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:43:22.115065    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c0dfef3-38b4-4195-a07c-acab5c17e850-model-cache\") pod \"1c0dfef3-38b4-4195-a07c-acab5c17e850\" (UID: \"1c0dfef3-38b4-4195-a07c-acab5c17e850\") "
May 25 11:43:22.115099 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:43:22.115104    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c0dfef3-38b4-4195-a07c-acab5c17e850-tls-certs\") pod \"1c0dfef3-38b4-4195-a07c-acab5c17e850\" (UID: \"1c0dfef3-38b4-4195-a07c-acab5c17e850\") "
May 25 11:43:22.115343 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:43:22.115135    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1c0dfef3-38b4-4195-a07c-acab5c17e850-kserve-provision-location\") pod \"1c0dfef3-38b4-4195-a07c-acab5c17e850\" (UID: \"1c0dfef3-38b4-4195-a07c-acab5c17e850\") "
May 25 11:43:22.115343 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:43:22.115158    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c0dfef3-38b4-4195-a07c-acab5c17e850-home\") pod \"1c0dfef3-38b4-4195-a07c-acab5c17e850\" (UID: \"1c0dfef3-38b4-4195-a07c-acab5c17e850\") "
May 25 11:43:22.115343 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:43:22.115183    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c0dfef3-38b4-4195-a07c-acab5c17e850-tmp-dir\") pod \"1c0dfef3-38b4-4195-a07c-acab5c17e850\" (UID: \"1c0dfef3-38b4-4195-a07c-acab5c17e850\") "
May 25 11:43:22.115343 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:43:22.115206    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r2hww\" (UniqueName: \"kubernetes.io/projected/1c0dfef3-38b4-4195-a07c-acab5c17e850-kube-api-access-r2hww\") pod \"1c0dfef3-38b4-4195-a07c-acab5c17e850\" (UID: \"1c0dfef3-38b4-4195-a07c-acab5c17e850\") "
May 25 11:43:22.115343 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:43:22.115313    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c0dfef3-38b4-4195-a07c-acab5c17e850-dshm\") pod \"1c0dfef3-38b4-4195-a07c-acab5c17e850\" (UID: \"1c0dfef3-38b4-4195-a07c-acab5c17e850\") "
May 25 11:43:22.115618 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:43:22.115455    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c0dfef3-38b4-4195-a07c-acab5c17e850-model-cache" (OuterVolumeSpecName: "model-cache") pod "1c0dfef3-38b4-4195-a07c-acab5c17e850" (UID: "1c0dfef3-38b4-4195-a07c-acab5c17e850"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:43:22.115699 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:43:22.115670    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/77cb6ea6-0019-4f5a-a219-72854cf6c824-tls-certs\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
May 25 11:43:22.115699 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:43:22.115696    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/77cb6ea6-0019-4f5a-a219-72854cf6c824-model-cache\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
May 25 11:43:22.115810 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:43:22.115710    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/77cb6ea6-0019-4f5a-a219-72854cf6c824-dshm\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
May 25 11:43:22.115810 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:43:22.115724    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/77cb6ea6-0019-4f5a-a219-72854cf6c824-tmp-dir\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
May 25 11:43:22.115810 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:43:22.115739    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tmgkb\" (UniqueName: \"kubernetes.io/projected/77cb6ea6-0019-4f5a-a219-72854cf6c824-kube-api-access-tmgkb\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
May 25 11:43:22.115810 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:43:22.115755    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c0dfef3-38b4-4195-a07c-acab5c17e850-model-cache\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
May 25 11:43:22.115810 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:43:22.115769    2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/77cb6ea6-0019-4f5a-a219-72854cf6c824-kserve-provision-location\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
May 25 11:43:22.115810 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:43:22.115782    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/77cb6ea6-0019-4f5a-a219-72854cf6c824-home\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
May 25 11:43:22.116642 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:43:22.116604    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c0dfef3-38b4-4195-a07c-acab5c17e850-home" (OuterVolumeSpecName: "home") pod "1c0dfef3-38b4-4195-a07c-acab5c17e850" (UID: "1c0dfef3-38b4-4195-a07c-acab5c17e850"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:43:22.118171 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:43:22.118132    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c0dfef3-38b4-4195-a07c-acab5c17e850-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1c0dfef3-38b4-4195-a07c-acab5c17e850" (UID: "1c0dfef3-38b4-4195-a07c-acab5c17e850"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 11:43:22.118304 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:43:22.118193    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c0dfef3-38b4-4195-a07c-acab5c17e850-kube-api-access-r2hww" (OuterVolumeSpecName: "kube-api-access-r2hww") pod "1c0dfef3-38b4-4195-a07c-acab5c17e850" (UID: "1c0dfef3-38b4-4195-a07c-acab5c17e850"). InnerVolumeSpecName "kube-api-access-r2hww". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 11:43:22.118304 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:43:22.118201    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c0dfef3-38b4-4195-a07c-acab5c17e850-dshm" (OuterVolumeSpecName: "dshm") pod "1c0dfef3-38b4-4195-a07c-acab5c17e850" (UID: "1c0dfef3-38b4-4195-a07c-acab5c17e850"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:43:22.129040 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:43:22.129003    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c0dfef3-38b4-4195-a07c-acab5c17e850-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1c0dfef3-38b4-4195-a07c-acab5c17e850" (UID: "1c0dfef3-38b4-4195-a07c-acab5c17e850"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:43:22.173371 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:43:22.173314    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c0dfef3-38b4-4195-a07c-acab5c17e850-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "1c0dfef3-38b4-4195-a07c-acab5c17e850" (UID: "1c0dfef3-38b4-4195-a07c-acab5c17e850"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:43:22.216757 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:43:22.216715    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c0dfef3-38b4-4195-a07c-acab5c17e850-dshm\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
May 25 11:43:22.216757 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:43:22.216747    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c0dfef3-38b4-4195-a07c-acab5c17e850-tls-certs\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
May 25 11:43:22.216757 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:43:22.216762    2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1c0dfef3-38b4-4195-a07c-acab5c17e850-kserve-provision-location\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
May 25 11:43:22.216757 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:43:22.216770    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c0dfef3-38b4-4195-a07c-acab5c17e850-home\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
May 25 11:43:22.217128 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:43:22.216780    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c0dfef3-38b4-4195-a07c-acab5c17e850-tmp-dir\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
May 25 11:43:22.217128 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:43:22.216789    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r2hww\" (UniqueName: \"kubernetes.io/projected/1c0dfef3-38b4-4195-a07c-acab5c17e850-kube-api-access-r2hww\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
May 25 11:43:22.449049 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:43:22.449012    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_custom-route-timeout-pd-test-kserve-6b47c648bc-kgnng_1c0dfef3-38b4-4195-a07c-acab5c17e850/main/0.log"
May 25 11:43:22.449760 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:43:22.449731    2576 generic.go:358] "Generic (PLEG): container finished" podID="1c0dfef3-38b4-4195-a07c-acab5c17e850" containerID="4f8533625cf9d1f67bd9e991c79e026513dd35dbc43158603a589b1f2dc54fc9" exitCode=137
May 25 11:43:22.449760 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:43:22.449755    2576 generic.go:358] "Generic (PLEG): container finished" podID="1c0dfef3-38b4-4195-a07c-acab5c17e850" containerID="6b34ee9d8e732ca40b6f96fa3f02559f62b2f50b0c96d9b4e7da362fcce8abe0" exitCode=0
May 25 11:43:22.449971 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:43:22.449820    2576 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6b47c648bc-kgnng"
May 25 11:43:22.449971 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:43:22.449828    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6b47c648bc-kgnng" event={"ID":"1c0dfef3-38b4-4195-a07c-acab5c17e850","Type":"ContainerDied","Data":"4f8533625cf9d1f67bd9e991c79e026513dd35dbc43158603a589b1f2dc54fc9"}
May 25 11:43:22.449971 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:43:22.449870    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6b47c648bc-kgnng" event={"ID":"1c0dfef3-38b4-4195-a07c-acab5c17e850","Type":"ContainerDied","Data":"6b34ee9d8e732ca40b6f96fa3f02559f62b2f50b0c96d9b4e7da362fcce8abe0"}
May 25 11:43:22.449971 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:43:22.449888    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6b47c648bc-kgnng" event={"ID":"1c0dfef3-38b4-4195-a07c-acab5c17e850","Type":"ContainerDied","Data":"564a5ea63cd4e8a3306ea79ab8bce9ae9f3586f2591f432de4c3f59f0b632bc2"}
May 25 11:43:22.449971 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:43:22.449909    2576 scope.go:117] "RemoveContainer" containerID="4f8533625cf9d1f67bd9e991c79e026513dd35dbc43158603a589b1f2dc54fc9"
May 25 11:43:22.451833 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:43:22.451802    2576 generic.go:358] "Generic (PLEG): container finished" podID="77cb6ea6-0019-4f5a-a219-72854cf6c824" containerID="838bbab20ca93f7029560b5dfadda21e257c2acd4c1b1deb3d37563855a85dfd" exitCode=137
May 25 11:43:22.451970 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:43:22.451868    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-545854866-cq97p" event={"ID":"77cb6ea6-0019-4f5a-a219-72854cf6c824","Type":"ContainerDied","Data":"838bbab20ca93f7029560b5dfadda21e257c2acd4c1b1deb3d37563855a85dfd"}
May 25 11:43:22.451970 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:43:22.451896    2576 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-545854866-cq97p"
May 25 11:43:22.452166 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:43:22.451897    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-545854866-cq97p" event={"ID":"77cb6ea6-0019-4f5a-a219-72854cf6c824","Type":"ContainerDied","Data":"8dc7123b3b49fa7e5031616fb63543de87d637742b515ed81be4583e0cdd5c1f"}
May 25 11:43:22.460121 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:43:22.460093    2576 scope.go:117] "RemoveContainer" containerID="b3bef6c9e7c5f08232bbb889469fe7839c656f687bf098a8832a0944a5a2cd19"
May 25 11:43:22.472408 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:43:22.472383    2576 scope.go:117] "RemoveContainer" containerID="6b34ee9d8e732ca40b6f96fa3f02559f62b2f50b0c96d9b4e7da362fcce8abe0"
May 25 11:43:22.481647 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:43:22.481621    2576 scope.go:117] "RemoveContainer" containerID="4f8533625cf9d1f67bd9e991c79e026513dd35dbc43158603a589b1f2dc54fc9"
May 25 11:43:22.482190 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:43:22.482167    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f8533625cf9d1f67bd9e991c79e026513dd35dbc43158603a589b1f2dc54fc9\": container with ID starting with 4f8533625cf9d1f67bd9e991c79e026513dd35dbc43158603a589b1f2dc54fc9 not found: ID does not exist" containerID="4f8533625cf9d1f67bd9e991c79e026513dd35dbc43158603a589b1f2dc54fc9"
May 25 11:43:22.482503 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:43:22.482473    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f8533625cf9d1f67bd9e991c79e026513dd35dbc43158603a589b1f2dc54fc9"} err="failed to get container status \"4f8533625cf9d1f67bd9e991c79e026513dd35dbc43158603a589b1f2dc54fc9\": rpc error: code = NotFound desc = could not find container \"4f8533625cf9d1f67bd9e991c79e026513dd35dbc43158603a589b1f2dc54fc9\": container with ID starting with 4f8533625cf9d1f67bd9e991c79e026513dd35dbc43158603a589b1f2dc54fc9 not found: ID does not exist"
May 25 11:43:22.482637 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:43:22.482623    2576 scope.go:117] "RemoveContainer" containerID="b3bef6c9e7c5f08232bbb889469fe7839c656f687bf098a8832a0944a5a2cd19"
May 25 11:43:22.482778 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:43:22.482492    2576 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6b47c648bc-kgnng"]
May 25 11:43:22.483328 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:43:22.483191    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3bef6c9e7c5f08232bbb889469fe7839c656f687bf098a8832a0944a5a2cd19\": container with ID starting with b3bef6c9e7c5f08232bbb889469fe7839c656f687bf098a8832a0944a5a2cd19 not found: ID does not exist" containerID="b3bef6c9e7c5f08232bbb889469fe7839c656f687bf098a8832a0944a5a2cd19"
May 25 11:43:22.483328 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:43:22.483221    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3bef6c9e7c5f08232bbb889469fe7839c656f687bf098a8832a0944a5a2cd19"} err="failed to get container status \"b3bef6c9e7c5f08232bbb889469fe7839c656f687bf098a8832a0944a5a2cd19\": rpc error: code = NotFound desc = could not find container \"b3bef6c9e7c5f08232bbb889469fe7839c656f687bf098a8832a0944a5a2cd19\": container with ID starting with b3bef6c9e7c5f08232bbb889469fe7839c656f687bf098a8832a0944a5a2cd19 not found: ID does not exist"
May 25 11:43:22.483328 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:43:22.483242    2576 scope.go:117] "RemoveContainer" containerID="6b34ee9d8e732ca40b6f96fa3f02559f62b2f50b0c96d9b4e7da362fcce8abe0"
May 25 11:43:22.483648 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:43:22.483623    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b34ee9d8e732ca40b6f96fa3f02559f62b2f50b0c96d9b4e7da362fcce8abe0\": container with ID starting with 6b34ee9d8e732ca40b6f96fa3f02559f62b2f50b0c96d9b4e7da362fcce8abe0 not found: ID does not exist" containerID="6b34ee9d8e732ca40b6f96fa3f02559f62b2f50b0c96d9b4e7da362fcce8abe0"
May 25 11:43:22.483720 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:43:22.483656    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b34ee9d8e732ca40b6f96fa3f02559f62b2f50b0c96d9b4e7da362fcce8abe0"} err="failed to get container status \"6b34ee9d8e732ca40b6f96fa3f02559f62b2f50b0c96d9b4e7da362fcce8abe0\": rpc error: code = NotFound desc = could not find container \"6b34ee9d8e732ca40b6f96fa3f02559f62b2f50b0c96d9b4e7da362fcce8abe0\": container with ID starting with 6b34ee9d8e732ca40b6f96fa3f02559f62b2f50b0c96d9b4e7da362fcce8abe0 not found: ID does not exist"
May 25 11:43:22.483720 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:43:22.483680    2576 scope.go:117] "RemoveContainer" containerID="4f8533625cf9d1f67bd9e991c79e026513dd35dbc43158603a589b1f2dc54fc9"
May 25 11:43:22.483951 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:43:22.483932    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f8533625cf9d1f67bd9e991c79e026513dd35dbc43158603a589b1f2dc54fc9"} err="failed to get container status \"4f8533625cf9d1f67bd9e991c79e026513dd35dbc43158603a589b1f2dc54fc9\": rpc error: code = NotFound desc = could not find container \"4f8533625cf9d1f67bd9e991c79e026513dd35dbc43158603a589b1f2dc54fc9\": container with ID starting with 4f8533625cf9d1f67bd9e991c79e026513dd35dbc43158603a589b1f2dc54fc9 not found: ID does not exist"
May 25 11:43:22.484009 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:43:22.483952    2576 scope.go:117] "RemoveContainer" containerID="b3bef6c9e7c5f08232bbb889469fe7839c656f687bf098a8832a0944a5a2cd19"
May 25 11:43:22.484223 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:43:22.484176    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3bef6c9e7c5f08232bbb889469fe7839c656f687bf098a8832a0944a5a2cd19"} err="failed to get container status \"b3bef6c9e7c5f08232bbb889469fe7839c656f687bf098a8832a0944a5a2cd19\": rpc error: code = NotFound desc = could not find container \"b3bef6c9e7c5f08232bbb889469fe7839c656f687bf098a8832a0944a5a2cd19\": container with ID starting with b3bef6c9e7c5f08232bbb889469fe7839c656f687bf098a8832a0944a5a2cd19 not found: ID does not exist"
May 25 11:43:22.484223 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:43:22.484223    2576 scope.go:117] "RemoveContainer" containerID="6b34ee9d8e732ca40b6f96fa3f02559f62b2f50b0c96d9b4e7da362fcce8abe0"
May 25 11:43:22.484527 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:43:22.484507    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b34ee9d8e732ca40b6f96fa3f02559f62b2f50b0c96d9b4e7da362fcce8abe0"} err="failed to get container status \"6b34ee9d8e732ca40b6f96fa3f02559f62b2f50b0c96d9b4e7da362fcce8abe0\": rpc error: code = NotFound desc = could not find container \"6b34ee9d8e732ca40b6f96fa3f02559f62b2f50b0c96d9b4e7da362fcce8abe0\": container with ID starting with 6b34ee9d8e732ca40b6f96fa3f02559f62b2f50b0c96d9b4e7da362fcce8abe0 not found: ID does not exist"
May 25 11:43:22.484619 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:43:22.484528    2576 scope.go:117] "RemoveContainer" containerID="838bbab20ca93f7029560b5dfadda21e257c2acd4c1b1deb3d37563855a85dfd"
May 25 11:43:22.489652 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:43:22.489625    2576 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6b47c648bc-kgnng"]
May 25 11:43:22.493913 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:43:22.493889    2576 scope.go:117] "RemoveContainer" containerID="96326c724e8dee4b11634d6fadda73fd76d303307bdccef68a13cf476a08c4c6"
May 25 11:43:22.501482 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:43:22.501448    2576 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-545854866-cq97p"]
May 25 11:43:22.508772 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:43:22.508736    2576 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-545854866-cq97p"]
May 25 11:43:22.568325 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:43:22.568046    2576 scope.go:117] "RemoveContainer" containerID="838bbab20ca93f7029560b5dfadda21e257c2acd4c1b1deb3d37563855a85dfd"
May 25 11:43:22.568528 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:43:22.568497    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"838bbab20ca93f7029560b5dfadda21e257c2acd4c1b1deb3d37563855a85dfd\": container with ID starting with 838bbab20ca93f7029560b5dfadda21e257c2acd4c1b1deb3d37563855a85dfd not found: ID does not exist" containerID="838bbab20ca93f7029560b5dfadda21e257c2acd4c1b1deb3d37563855a85dfd"
May 25 11:43:22.568601 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:43:22.568541    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"838bbab20ca93f7029560b5dfadda21e257c2acd4c1b1deb3d37563855a85dfd"} err="failed to get container status \"838bbab20ca93f7029560b5dfadda21e257c2acd4c1b1deb3d37563855a85dfd\": rpc error: code = NotFound desc = could not find container \"838bbab20ca93f7029560b5dfadda21e257c2acd4c1b1deb3d37563855a85dfd\": container with ID starting with 838bbab20ca93f7029560b5dfadda21e257c2acd4c1b1deb3d37563855a85dfd not found: ID does not exist"
May 25 11:43:22.568601 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:43:22.568576    2576 scope.go:117] "RemoveContainer" containerID="96326c724e8dee4b11634d6fadda73fd76d303307bdccef68a13cf476a08c4c6"
May 25 11:43:22.568875 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:43:22.568853    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96326c724e8dee4b11634d6fadda73fd76d303307bdccef68a13cf476a08c4c6\": container with ID starting with 96326c724e8dee4b11634d6fadda73fd76d303307bdccef68a13cf476a08c4c6 not found: ID does not exist" containerID="96326c724e8dee4b11634d6fadda73fd76d303307bdccef68a13cf476a08c4c6"
May 25 11:43:22.568961 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:43:22.568887    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96326c724e8dee4b11634d6fadda73fd76d303307bdccef68a13cf476a08c4c6"} err="failed to get container status \"96326c724e8dee4b11634d6fadda73fd76d303307bdccef68a13cf476a08c4c6\": rpc error: code = NotFound desc = could not find container \"96326c724e8dee4b11634d6fadda73fd76d303307bdccef68a13cf476a08c4c6\": container with ID starting with 96326c724e8dee4b11634d6fadda73fd76d303307bdccef68a13cf476a08c4c6 not found: ID does not exist"
May 25 11:43:23.098439 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:43:23.098401    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1c0dfef3-38b4-4195-a07c-acab5c17e850" path="/var/lib/kubelet/pods/1c0dfef3-38b4-4195-a07c-acab5c17e850/volumes"
May 25 11:43:23.098891 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:43:23.098873    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="77cb6ea6-0019-4f5a-a219-72854cf6c824" path="/var/lib/kubelet/pods/77cb6ea6-0019-4f5a-a219-72854cf6c824/volumes"
May 25 11:43:31.495228 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:43:31.495165    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-799dc8b8c-8w6n5" podUID="e96af524-82a6-41ef-8bf2-8a1bf59bc33f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8001/health\": dial tcp 10.133.0.38:8001: connect: connection refused"
May 25 11:43:31.520390 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:43:31.520346    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-78459cb79d-92lfb" podUID="887d4410-aee9-44bc-99ee-9fa600b7b6ac" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/health\": dial tcp 10.133.0.39:8000: connect: connection refused"
May 25 11:43:41.495352 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:43:41.495303    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-799dc8b8c-8w6n5" podUID="e96af524-82a6-41ef-8bf2-8a1bf59bc33f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8001/health\": dial tcp 10.133.0.38:8001: connect: connection refused"
May 25 11:43:41.521128 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:43:41.521080    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-78459cb79d-92lfb" podUID="887d4410-aee9-44bc-99ee-9fa600b7b6ac" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/health\": dial tcp 10.133.0.39:8000: connect: connection refused"
May 25 11:43:51.495980 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:43:51.495938    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-799dc8b8c-8w6n5" podUID="e96af524-82a6-41ef-8bf2-8a1bf59bc33f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8001/health\": dial tcp 10.133.0.38:8001: connect: connection refused"
May 25 11:43:51.519509 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:43:51.519459    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-78459cb79d-92lfb" podUID="887d4410-aee9-44bc-99ee-9fa600b7b6ac" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/health\": dial tcp 10.133.0.39:8000: connect: connection refused"
May 25 11:44:01.496087 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:44:01.495988    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-799dc8b8c-8w6n5" podUID="e96af524-82a6-41ef-8bf2-8a1bf59bc33f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8001/health\": dial tcp 10.133.0.38:8001: connect: connection refused"
May 25 11:44:01.519724 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:44:01.519680    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-78459cb79d-92lfb" podUID="887d4410-aee9-44bc-99ee-9fa600b7b6ac" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/health\": dial tcp 10.133.0.39:8000: connect: connection refused"
May 25 11:44:11.495033 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:44:11.494978    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-799dc8b8c-8w6n5" podUID="e96af524-82a6-41ef-8bf2-8a1bf59bc33f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8001/health\": dial tcp 10.133.0.38:8001: connect: connection refused"
May 25 11:44:11.520029 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:44:11.519981    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-78459cb79d-92lfb" podUID="887d4410-aee9-44bc-99ee-9fa600b7b6ac" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/health\": dial tcp 10.133.0.39:8000: connect: connection refused"
May 25 11:44:21.494972 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:44:21.494871    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-799dc8b8c-8w6n5" podUID="e96af524-82a6-41ef-8bf2-8a1bf59bc33f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8001/health\": dial tcp 10.133.0.38:8001: connect: connection refused"
May 25 11:44:21.520450 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:44:21.520418    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-78459cb79d-92lfb" podUID="887d4410-aee9-44bc-99ee-9fa600b7b6ac" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/health\": dial tcp 10.133.0.39:8000: connect: connection refused"
May 25 11:44:31.495438 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:44:31.495391    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-799dc8b8c-8w6n5" podUID="e96af524-82a6-41ef-8bf2-8a1bf59bc33f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8001/health\": dial tcp 10.133.0.38:8001: connect: connection refused"
May 25 11:44:31.520492 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:44:31.520442    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-78459cb79d-92lfb" podUID="887d4410-aee9-44bc-99ee-9fa600b7b6ac" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/health\": dial tcp 10.133.0.39:8000: connect: connection refused"
May 25 11:44:41.495526 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:44:41.495473    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-799dc8b8c-8w6n5" podUID="e96af524-82a6-41ef-8bf2-8a1bf59bc33f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8001/health\": dial tcp 10.133.0.38:8001: connect: connection refused"
May 25 11:44:41.520072 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:44:41.520036    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-78459cb79d-92lfb" podUID="887d4410-aee9-44bc-99ee-9fa600b7b6ac" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/health\": dial tcp 10.133.0.39:8000: connect: connection refused"
May 25 11:44:51.495957 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:44:51.495894    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-799dc8b8c-8w6n5" podUID="e96af524-82a6-41ef-8bf2-8a1bf59bc33f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8001/health\": dial tcp 10.133.0.38:8001: connect: connection refused"
May 25 11:44:51.519751 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:44:51.519709    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-78459cb79d-92lfb" podUID="887d4410-aee9-44bc-99ee-9fa600b7b6ac" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/health\": dial tcp 10.133.0.39:8000: connect: connection refused"
May 25 11:45:01.495991 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:45:01.495936    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-799dc8b8c-8w6n5" podUID="e96af524-82a6-41ef-8bf2-8a1bf59bc33f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8001/health\": dial tcp 10.133.0.38:8001: connect: connection refused"
May 25 11:45:01.520385 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:45:01.520346    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-78459cb79d-92lfb" podUID="887d4410-aee9-44bc-99ee-9fa600b7b6ac" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/health\": dial tcp 10.133.0.39:8000: connect: connection refused"
May 25 11:45:11.495363 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:45:11.495322    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-799dc8b8c-8w6n5" podUID="e96af524-82a6-41ef-8bf2-8a1bf59bc33f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8001/health\": dial tcp 10.133.0.38:8001: connect: connection refused"
May 25 11:45:11.535626 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:45:11.535594    2576 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-78459cb79d-92lfb"
May 25 11:45:11.543419 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:45:11.543399    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-78459cb79d-92lfb"
May 25 11:45:21.504447 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:45:21.504418    2576 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-799dc8b8c-8w6n5"
May 25 11:45:21.516546 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:45:21.516524    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-799dc8b8c-8w6n5"
May 25 11:46:55.804424 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:46:55.804390    2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/conv-test-criticality-kserve-7c79ff4697-g6zqz"]
May 25 11:46:55.804953 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:46:55.804773    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="77cb6ea6-0019-4f5a-a219-72854cf6c824" containerName="main"
May 25 11:46:55.804953 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:46:55.804787    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="77cb6ea6-0019-4f5a-a219-72854cf6c824" containerName="main"
May 25 11:46:55.804953 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:46:55.804799    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c0dfef3-38b4-4195-a07c-acab5c17e850" containerName="llm-d-routing-sidecar"
May 25 11:46:55.804953 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:46:55.804806    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c0dfef3-38b4-4195-a07c-acab5c17e850" containerName="llm-d-routing-sidecar"
May 25 11:46:55.804953 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:46:55.804823    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c0dfef3-38b4-4195-a07c-acab5c17e850" containerName="storage-initializer"
May 25 11:46:55.804953 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:46:55.804832    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c0dfef3-38b4-4195-a07c-acab5c17e850" containerName="storage-initializer"
May 25 11:46:55.804953 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:46:55.804845    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="77cb6ea6-0019-4f5a-a219-72854cf6c824" containerName="storage-initializer"
May 25 11:46:55.804953 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:46:55.804850    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="77cb6ea6-0019-4f5a-a219-72854cf6c824" containerName="storage-initializer"
May 25 11:46:55.804953 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:46:55.804857    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c0dfef3-38b4-4195-a07c-acab5c17e850" containerName="main"
May 25 11:46:55.804953 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:46:55.804861    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c0dfef3-38b4-4195-a07c-acab5c17e850" containerName="main"
May 25 11:46:55.804953 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:46:55.804917    2576 memory_manager.go:356] "RemoveStaleState removing state" podUID="77cb6ea6-0019-4f5a-a219-72854cf6c824" containerName="main"
May 25 11:46:55.804953 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:46:55.804926    2576 memory_manager.go:356] "RemoveStaleState removing state" podUID="1c0dfef3-38b4-4195-a07c-acab5c17e850" containerName="llm-d-routing-sidecar"
May 25 11:46:55.804953 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:46:55.804934    2576 memory_manager.go:356] "RemoveStaleState removing state" podUID="1c0dfef3-38b4-4195-a07c-acab5c17e850" containerName="main"
May 25 11:46:55.808093 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:46:55.808078    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/conv-test-criticality-kserve-7c79ff4697-g6zqz"
May 25 11:46:55.810571 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:46:55.810550    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"conv-test-criticality-kserve-self-signed-certs\""
May 25 11:46:55.818727 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:46:55.818706    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/conv-test-criticality-kserve-7c79ff4697-g6zqz"]
May 25 11:46:55.891449 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:46:55.891418    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad589270-6564-4cdd-a936-e645856ca27c-tmp-dir\") pod \"conv-test-criticality-kserve-7c79ff4697-g6zqz\" (UID: \"ad589270-6564-4cdd-a936-e645856ca27c\") " pod="kserve-ci-e2e-test/conv-test-criticality-kserve-7c79ff4697-g6zqz"
May 25 11:46:55.891630 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:46:55.891457    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad589270-6564-4cdd-a936-e645856ca27c-tls-certs\") pod \"conv-test-criticality-kserve-7c79ff4697-g6zqz\" (UID: \"ad589270-6564-4cdd-a936-e645856ca27c\") " pod="kserve-ci-e2e-test/conv-test-criticality-kserve-7c79ff4697-g6zqz"
May 25 11:46:55.891630 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:46:55.891505    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ad589270-6564-4cdd-a936-e645856ca27c-kserve-provision-location\") pod \"conv-test-criticality-kserve-7c79ff4697-g6zqz\" (UID: \"ad589270-6564-4cdd-a936-e645856ca27c\") " pod="kserve-ci-e2e-test/conv-test-criticality-kserve-7c79ff4697-g6zqz"
May 25 11:46:55.891630 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:46:55.891542    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad589270-6564-4cdd-a936-e645856ca27c-model-cache\") pod \"conv-test-criticality-kserve-7c79ff4697-g6zqz\" (UID: \"ad589270-6564-4cdd-a936-e645856ca27c\") " pod="kserve-ci-e2e-test/conv-test-criticality-kserve-7c79ff4697-g6zqz"
May 25 11:46:55.891630 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:46:55.891564    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad589270-6564-4cdd-a936-e645856ca27c-home\") pod \"conv-test-criticality-kserve-7c79ff4697-g6zqz\" (UID: \"ad589270-6564-4cdd-a936-e645856ca27c\") " pod="kserve-ci-e2e-test/conv-test-criticality-kserve-7c79ff4697-g6zqz"
May 25 11:46:55.891630 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:46:55.891612    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-678tk\" (UniqueName: \"kubernetes.io/projected/ad589270-6564-4cdd-a936-e645856ca27c-kube-api-access-678tk\") pod \"conv-test-criticality-kserve-7c79ff4697-g6zqz\" (UID: \"ad589270-6564-4cdd-a936-e645856ca27c\") " pod="kserve-ci-e2e-test/conv-test-criticality-kserve-7c79ff4697-g6zqz"
May 25 11:46:55.891824 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:46:55.891693    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad589270-6564-4cdd-a936-e645856ca27c-dshm\") pod \"conv-test-criticality-kserve-7c79ff4697-g6zqz\" (UID: \"ad589270-6564-4cdd-a936-e645856ca27c\") " pod="kserve-ci-e2e-test/conv-test-criticality-kserve-7c79ff4697-g6zqz"
May 25 11:46:55.992477 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:46:55.992442    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ad589270-6564-4cdd-a936-e645856ca27c-kserve-provision-location\") pod \"conv-test-criticality-kserve-7c79ff4697-g6zqz\" (UID: \"ad589270-6564-4cdd-a936-e645856ca27c\") " pod="kserve-ci-e2e-test/conv-test-criticality-kserve-7c79ff4697-g6zqz"
May 25 11:46:55.992670 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:46:55.992487    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad589270-6564-4cdd-a936-e645856ca27c-model-cache\") pod \"conv-test-criticality-kserve-7c79ff4697-g6zqz\" (UID: \"ad589270-6564-4cdd-a936-e645856ca27c\") " pod="kserve-ci-e2e-test/conv-test-criticality-kserve-7c79ff4697-g6zqz"
May 25 11:46:55.992670 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:46:55.992507    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad589270-6564-4cdd-a936-e645856ca27c-home\") pod \"conv-test-criticality-kserve-7c79ff4697-g6zqz\" (UID: \"ad589270-6564-4cdd-a936-e645856ca27c\") " pod="kserve-ci-e2e-test/conv-test-criticality-kserve-7c79ff4697-g6zqz"
May 25 11:46:55.992670 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:46:55.992526    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-678tk\" (UniqueName: \"kubernetes.io/projected/ad589270-6564-4cdd-a936-e645856ca27c-kube-api-access-678tk\") pod \"conv-test-criticality-kserve-7c79ff4697-g6zqz\" (UID: \"ad589270-6564-4cdd-a936-e645856ca27c\") " pod="kserve-ci-e2e-test/conv-test-criticality-kserve-7c79ff4697-g6zqz"
May 25 11:46:55.992670 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:46:55.992564    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad589270-6564-4cdd-a936-e645856ca27c-dshm\") pod \"conv-test-criticality-kserve-7c79ff4697-g6zqz\" (UID: \"ad589270-6564-4cdd-a936-e645856ca27c\") " pod="kserve-ci-e2e-test/conv-test-criticality-kserve-7c79ff4697-g6zqz"
May 25 11:46:55.992670 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:46:55.992624    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad589270-6564-4cdd-a936-e645856ca27c-tmp-dir\") pod \"conv-test-criticality-kserve-7c79ff4697-g6zqz\" (UID: \"ad589270-6564-4cdd-a936-e645856ca27c\") " pod="kserve-ci-e2e-test/conv-test-criticality-kserve-7c79ff4697-g6zqz"
May 25 11:46:55.992968 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:46:55.992746    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad589270-6564-4cdd-a936-e645856ca27c-tls-certs\") pod \"conv-test-criticality-kserve-7c79ff4697-g6zqz\" (UID: \"ad589270-6564-4cdd-a936-e645856ca27c\") " pod="kserve-ci-e2e-test/conv-test-criticality-kserve-7c79ff4697-g6zqz"
May 25 11:46:55.992968 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:46:55.992902    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ad589270-6564-4cdd-a936-e645856ca27c-kserve-provision-location\") pod \"conv-test-criticality-kserve-7c79ff4697-g6zqz\" (UID: \"ad589270-6564-4cdd-a936-e645856ca27c\") " pod="kserve-ci-e2e-test/conv-test-criticality-kserve-7c79ff4697-g6zqz"
May 25 11:46:55.992968 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:46:55.992937    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad589270-6564-4cdd-a936-e645856ca27c-home\") pod \"conv-test-criticality-kserve-7c79ff4697-g6zqz\" (UID: \"ad589270-6564-4cdd-a936-e645856ca27c\") " pod="kserve-ci-e2e-test/conv-test-criticality-kserve-7c79ff4697-g6zqz"
May 25 11:46:55.993071 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:46:55.992967    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad589270-6564-4cdd-a936-e645856ca27c-model-cache\") pod \"conv-test-criticality-kserve-7c79ff4697-g6zqz\" (UID: \"ad589270-6564-4cdd-a936-e645856ca27c\") " pod="kserve-ci-e2e-test/conv-test-criticality-kserve-7c79ff4697-g6zqz"
May 25 11:46:55.993071 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:46:55.993020    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad589270-6564-4cdd-a936-e645856ca27c-tmp-dir\") pod \"conv-test-criticality-kserve-7c79ff4697-g6zqz\" (UID: \"ad589270-6564-4cdd-a936-e645856ca27c\") " pod="kserve-ci-e2e-test/conv-test-criticality-kserve-7c79ff4697-g6zqz"
May 25 11:46:55.995159 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:46:55.995131    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad589270-6564-4cdd-a936-e645856ca27c-dshm\") pod \"conv-test-criticality-kserve-7c79ff4697-g6zqz\" (UID: \"ad589270-6564-4cdd-a936-e645856ca27c\") " pod="kserve-ci-e2e-test/conv-test-criticality-kserve-7c79ff4697-g6zqz"
May 25 11:46:55.995743 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:46:55.995570    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad589270-6564-4cdd-a936-e645856ca27c-tls-certs\") pod \"conv-test-criticality-kserve-7c79ff4697-g6zqz\" (UID: \"ad589270-6564-4cdd-a936-e645856ca27c\") " pod="kserve-ci-e2e-test/conv-test-criticality-kserve-7c79ff4697-g6zqz"
May 25 11:46:56.001421 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:46:56.001393    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-678tk\" (UniqueName: \"kubernetes.io/projected/ad589270-6564-4cdd-a936-e645856ca27c-kube-api-access-678tk\") pod \"conv-test-criticality-kserve-7c79ff4697-g6zqz\" (UID: \"ad589270-6564-4cdd-a936-e645856ca27c\") " pod="kserve-ci-e2e-test/conv-test-criticality-kserve-7c79ff4697-g6zqz"
May 25 11:46:56.118783 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:46:56.118704    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/conv-test-criticality-kserve-7c79ff4697-g6zqz"
May 25 11:46:56.281176 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:46:56.281153    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/conv-test-criticality-kserve-7c79ff4697-g6zqz"]
May 25 11:46:56.283626 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:46:56.283598    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad589270_6564_4cdd_a936_e645856ca27c.slice/crio-8a6238c47367a8c513a1d2ba8daf0b0b81f486a18f08af7a4656370d706bc18c WatchSource:0}: Error finding container 8a6238c47367a8c513a1d2ba8daf0b0b81f486a18f08af7a4656370d706bc18c: Status 404 returned error can't find the container with id 8a6238c47367a8c513a1d2ba8daf0b0b81f486a18f08af7a4656370d706bc18c
May 25 11:46:57.214075 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:46:57.214032    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/conv-test-criticality-kserve-7c79ff4697-g6zqz" event={"ID":"ad589270-6564-4cdd-a936-e645856ca27c","Type":"ContainerStarted","Data":"357a749bc4aacab371ac072280e5589af1871a2a271b33cf6331ec4c125f8c23"}
May 25 11:46:57.214075 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:46:57.214077    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/conv-test-criticality-kserve-7c79ff4697-g6zqz" event={"ID":"ad589270-6564-4cdd-a936-e645856ca27c","Type":"ContainerStarted","Data":"8a6238c47367a8c513a1d2ba8daf0b0b81f486a18f08af7a4656370d706bc18c"}
May 25 11:47:00.225749 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:47:00.225714    2576 generic.go:358] "Generic (PLEG): container finished" podID="ad589270-6564-4cdd-a936-e645856ca27c" containerID="357a749bc4aacab371ac072280e5589af1871a2a271b33cf6331ec4c125f8c23" exitCode=0
May 25 11:47:00.226105 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:47:00.225785    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/conv-test-criticality-kserve-7c79ff4697-g6zqz" event={"ID":"ad589270-6564-4cdd-a936-e645856ca27c","Type":"ContainerDied","Data":"357a749bc4aacab371ac072280e5589af1871a2a271b33cf6331ec4c125f8c23"}
May 25 11:47:01.231096 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:47:01.231062    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/conv-test-criticality-kserve-7c79ff4697-g6zqz" event={"ID":"ad589270-6564-4cdd-a936-e645856ca27c","Type":"ContainerStarted","Data":"1135587c9ed5c0967dff249c1977243ae12238214bca4e11a83de03f01e28f00"}
May 25 11:47:01.252893 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:47:01.252838    2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/conv-test-criticality-kserve-7c79ff4697-g6zqz" podStartSLOduration=6.2528214 podStartE2EDuration="6.2528214s" podCreationTimestamp="2026-05-25 11:46:55 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 11:47:01.251115964 +0000 UTC m=+1766.768847420" watchObservedRunningTime="2026-05-25 11:47:01.2528214 +0000 UTC m=+1766.770552857"
May 25 11:47:06.119036 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:47:06.119000    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/conv-test-criticality-kserve-7c79ff4697-g6zqz"
May 25 11:47:06.119476 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:47:06.119054    2576 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/conv-test-criticality-kserve-7c79ff4697-g6zqz"
May 25 11:47:06.120502 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:47:06.120469    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/conv-test-criticality-kserve-7c79ff4697-g6zqz" podUID="ad589270-6564-4cdd-a936-e645856ca27c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/health\": dial tcp 10.133.0.40:8000: connect: connection refused"
May 25 11:47:16.119625 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:47:16.119583    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/conv-test-criticality-kserve-7c79ff4697-g6zqz" podUID="ad589270-6564-4cdd-a936-e645856ca27c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/health\": dial tcp 10.133.0.40:8000: connect: connection refused"
May 25 11:47:17.366679 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:47:17.366595    2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-5449d77-tm4cr"]
May 25 11:47:17.379812 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:47:17.379787    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5449d77-tm4cr"
May 25 11:47:17.383111 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:47:17.383090    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"stop-feature-test-kserve-self-signed-certs\""
May 25 11:47:17.389997 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:47:17.389972    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-5449d77-tm4cr"]
May 25 11:47:17.499685 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:47:17.499649    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aacdbcbe-b12e-4969-9cfe-586811191878-home\") pod \"stop-feature-test-kserve-5449d77-tm4cr\" (UID: \"aacdbcbe-b12e-4969-9cfe-586811191878\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5449d77-tm4cr"
May 25 11:47:17.499897 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:47:17.499792    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aacdbcbe-b12e-4969-9cfe-586811191878-dshm\") pod \"stop-feature-test-kserve-5449d77-tm4cr\" (UID: \"aacdbcbe-b12e-4969-9cfe-586811191878\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5449d77-tm4cr"
May 25 11:47:17.499897 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:47:17.499821    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aacdbcbe-b12e-4969-9cfe-586811191878-tls-certs\") pod \"stop-feature-test-kserve-5449d77-tm4cr\" (UID: \"aacdbcbe-b12e-4969-9cfe-586811191878\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5449d77-tm4cr"
May 25 11:47:17.499897 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:47:17.499846    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aacdbcbe-b12e-4969-9cfe-586811191878-model-cache\") pod \"stop-feature-test-kserve-5449d77-tm4cr\" (UID: \"aacdbcbe-b12e-4969-9cfe-586811191878\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5449d77-tm4cr"
May 25 11:47:17.499897 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:47:17.499863    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aacdbcbe-b12e-4969-9cfe-586811191878-tmp-dir\") pod \"stop-feature-test-kserve-5449d77-tm4cr\" (UID: \"aacdbcbe-b12e-4969-9cfe-586811191878\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5449d77-tm4cr"
May 25 11:47:17.499897 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:47:17.499888    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/aacdbcbe-b12e-4969-9cfe-586811191878-kserve-provision-location\") pod \"stop-feature-test-kserve-5449d77-tm4cr\" (UID: \"aacdbcbe-b12e-4969-9cfe-586811191878\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5449d77-tm4cr"
May 25 11:47:17.500161 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:47:17.499916    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crtvg\" (UniqueName: \"kubernetes.io/projected/aacdbcbe-b12e-4969-9cfe-586811191878-kube-api-access-crtvg\") pod \"stop-feature-test-kserve-5449d77-tm4cr\" (UID: \"aacdbcbe-b12e-4969-9cfe-586811191878\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5449d77-tm4cr"
May 25 11:47:17.600688 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:47:17.600643    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aacdbcbe-b12e-4969-9cfe-586811191878-dshm\") pod \"stop-feature-test-kserve-5449d77-tm4cr\" (UID: \"aacdbcbe-b12e-4969-9cfe-586811191878\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5449d77-tm4cr"
May 25 11:47:17.600688 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:47:17.600683    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aacdbcbe-b12e-4969-9cfe-586811191878-tls-certs\") pod \"stop-feature-test-kserve-5449d77-tm4cr\" (UID: \"aacdbcbe-b12e-4969-9cfe-586811191878\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5449d77-tm4cr"
May 25 11:47:17.600944 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:47:17.600705    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aacdbcbe-b12e-4969-9cfe-586811191878-model-cache\") pod \"stop-feature-test-kserve-5449d77-tm4cr\" (UID: \"aacdbcbe-b12e-4969-9cfe-586811191878\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5449d77-tm4cr"
May 25 11:47:17.600944 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:47:17.600722    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aacdbcbe-b12e-4969-9cfe-586811191878-tmp-dir\") pod \"stop-feature-test-kserve-5449d77-tm4cr\" (UID: \"aacdbcbe-b12e-4969-9cfe-586811191878\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5449d77-tm4cr"
May 25 11:47:17.600944 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:47:17.600738    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/aacdbcbe-b12e-4969-9cfe-586811191878-kserve-provision-location\") pod \"stop-feature-test-kserve-5449d77-tm4cr\" (UID: \"aacdbcbe-b12e-4969-9cfe-586811191878\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5449d77-tm4cr"
May 25 11:47:17.600944 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:47:17.600763    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-crtvg\" (UniqueName: \"kubernetes.io/projected/aacdbcbe-b12e-4969-9cfe-586811191878-kube-api-access-crtvg\") pod \"stop-feature-test-kserve-5449d77-tm4cr\" (UID: \"aacdbcbe-b12e-4969-9cfe-586811191878\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5449d77-tm4cr"
May 25 11:47:17.600944 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:47:17.600794    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aacdbcbe-b12e-4969-9cfe-586811191878-home\") pod \"stop-feature-test-kserve-5449d77-tm4cr\" (UID: \"aacdbcbe-b12e-4969-9cfe-586811191878\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5449d77-tm4cr"
May 25 11:47:17.601176 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:47:17.601126    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aacdbcbe-b12e-4969-9cfe-586811191878-model-cache\") pod \"stop-feature-test-kserve-5449d77-tm4cr\" (UID: \"aacdbcbe-b12e-4969-9cfe-586811191878\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5449d77-tm4cr"
May 25 11:47:17.601226 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:47:17.601175    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aacdbcbe-b12e-4969-9cfe-586811191878-tmp-dir\") pod \"stop-feature-test-kserve-5449d77-tm4cr\" (UID: \"aacdbcbe-b12e-4969-9cfe-586811191878\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5449d77-tm4cr"
May 25 11:47:17.601308 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:47:17.601222    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/aacdbcbe-b12e-4969-9cfe-586811191878-kserve-provision-location\") pod \"stop-feature-test-kserve-5449d77-tm4cr\" (UID: \"aacdbcbe-b12e-4969-9cfe-586811191878\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5449d77-tm4cr"
May 25 11:47:17.601420 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:47:17.601402    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aacdbcbe-b12e-4969-9cfe-586811191878-home\") pod \"stop-feature-test-kserve-5449d77-tm4cr\" (UID: \"aacdbcbe-b12e-4969-9cfe-586811191878\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5449d77-tm4cr"
May 25 11:47:17.603359 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:47:17.603323    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aacdbcbe-b12e-4969-9cfe-586811191878-dshm\") pod \"stop-feature-test-kserve-5449d77-tm4cr\" (UID: \"aacdbcbe-b12e-4969-9cfe-586811191878\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5449d77-tm4cr"
May 25 11:47:17.603537 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:47:17.603516    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aacdbcbe-b12e-4969-9cfe-586811191878-tls-certs\") pod \"stop-feature-test-kserve-5449d77-tm4cr\" (UID: \"aacdbcbe-b12e-4969-9cfe-586811191878\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5449d77-tm4cr"
May 25 11:47:17.613695 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:47:17.613664    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-crtvg\" (UniqueName: \"kubernetes.io/projected/aacdbcbe-b12e-4969-9cfe-586811191878-kube-api-access-crtvg\") pod \"stop-feature-test-kserve-5449d77-tm4cr\" (UID: \"aacdbcbe-b12e-4969-9cfe-586811191878\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5449d77-tm4cr"
May 25 11:47:17.692114 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:47:17.692070    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5449d77-tm4cr"
May 25 11:47:17.853796 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:47:17.853759    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-5449d77-tm4cr"]
May 25 11:47:17.855101 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:47:17.855077    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaacdbcbe_b12e_4969_9cfe_586811191878.slice/crio-7d9a90da8986443583dc9365efe51bb28f4c9957d6341143dbd19d4916d737b3 WatchSource:0}: Error finding container 7d9a90da8986443583dc9365efe51bb28f4c9957d6341143dbd19d4916d737b3: Status 404 returned error can't find the container with id 7d9a90da8986443583dc9365efe51bb28f4c9957d6341143dbd19d4916d737b3
May 25 11:47:18.289708 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:47:18.289670    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5449d77-tm4cr" event={"ID":"aacdbcbe-b12e-4969-9cfe-586811191878","Type":"ContainerStarted","Data":"040b762224fcfa7e740f1a88d058ff4f7a4ed8007612d1ddef1786617e90eb36"}
May 25 11:47:18.289708 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:47:18.289712    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5449d77-tm4cr" event={"ID":"aacdbcbe-b12e-4969-9cfe-586811191878","Type":"ContainerStarted","Data":"7d9a90da8986443583dc9365efe51bb28f4c9957d6341143dbd19d4916d737b3"}
May 25 11:47:19.106194 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:47:19.106140    2576 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/conv-test-criticality-kserve-7c79ff4697-g6zqz"]
May 25 11:47:19.106803 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:47:19.106531    2576 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/conv-test-criticality-kserve-7c79ff4697-g6zqz" podUID="ad589270-6564-4cdd-a936-e645856ca27c" containerName="main" containerID="cri-o://1135587c9ed5c0967dff249c1977243ae12238214bca4e11a83de03f01e28f00" gracePeriod=30
May 25 11:47:35.154390 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:47:35.154360    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h6b5t_12b45353-8444-41fc-a8ed-fb691764eb63/ovn-acl-logging/0.log"
May 25 11:47:35.156133 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:47:35.156113    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h6b5t_12b45353-8444-41fc-a8ed-fb691764eb63/ovn-acl-logging/0.log"
May 25 11:47:40.617819 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:47:40.617783    2576 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-78459cb79d-92lfb"]
May 25 11:47:40.618362 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:47:40.618152    2576 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-78459cb79d-92lfb" podUID="887d4410-aee9-44bc-99ee-9fa600b7b6ac" containerName="main" containerID="cri-o://bd4f18fb0669d2bc012182ce6c58348fb7fdd977ce0815d73a782ca0e48bc728" gracePeriod=30
May 25 11:47:40.635330 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:47:40.635266    2576 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-799dc8b8c-8w6n5"]
May 25 11:47:40.635825 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:47:40.635792    2576 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-799dc8b8c-8w6n5" podUID="e96af524-82a6-41ef-8bf2-8a1bf59bc33f" containerName="main" containerID="cri-o://b60aa49e873e79bbea2ce5a73b8123ed1f3fcac0abc08b5393ed0aaf61aa767d" gracePeriod=30
May 25 11:47:47.300516 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:47:47.300479    2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b5bb44845g5xgx"]
May 25 11:47:47.305636 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:47:47.305607    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b5bb44845g5xgx"
May 25 11:47:47.307902 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:47:47.307876    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv8de1d74aab16d9cabd8b5aafeb5248e8-kserve-self-signed-certs\""
May 25 11:47:47.315304 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:47:47.315257    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b5bb44845g5xgx"]
May 25 11:47:47.406405 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:47:47.406369    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce4cff5d-2132-4b5c-9aea-790191e87b00-dshm\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b5bb44845g5xgx\" (UID: \"ce4cff5d-2132-4b5c-9aea-790191e87b00\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b5bb44845g5xgx"
May 25 11:47:47.406405 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:47:47.406413    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjvzz\" (UniqueName: \"kubernetes.io/projected/ce4cff5d-2132-4b5c-9aea-790191e87b00-kube-api-access-qjvzz\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b5bb44845g5xgx\" (UID: \"ce4cff5d-2132-4b5c-9aea-790191e87b00\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b5bb44845g5xgx"
May 25 11:47:47.406752 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:47:47.406505    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ce4cff5d-2132-4b5c-9aea-790191e87b00-home\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b5bb44845g5xgx\" (UID: \"ce4cff5d-2132-4b5c-9aea-790191e87b00\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b5bb44845g5xgx"
May 25 11:47:47.406752 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:47:47.406568    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce4cff5d-2132-4b5c-9aea-790191e87b00-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b5bb44845g5xgx\" (UID: \"ce4cff5d-2132-4b5c-9aea-790191e87b00\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b5bb44845g5xgx"
May 25 11:47:47.406752 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:47:47.406673    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce4cff5d-2132-4b5c-9aea-790191e87b00-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b5bb44845g5xgx\" (UID: \"ce4cff5d-2132-4b5c-9aea-790191e87b00\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b5bb44845g5xgx"
May 25 11:47:47.406752 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:47:47.406704    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ce4cff5d-2132-4b5c-9aea-790191e87b00-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b5bb44845g5xgx\" (UID: \"ce4cff5d-2132-4b5c-9aea-790191e87b00\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b5bb44845g5xgx"
May 25 11:47:47.406752 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:47:47.406735    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce4cff5d-2132-4b5c-9aea-790191e87b00-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b5bb44845g5xgx\" (UID: \"ce4cff5d-2132-4b5c-9aea-790191e87b00\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b5bb44845g5xgx"
May 25 11:47:47.507263 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:47:47.507226    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce4cff5d-2132-4b5c-9aea-790191e87b00-dshm\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b5bb44845g5xgx\" (UID: \"ce4cff5d-2132-4b5c-9aea-790191e87b00\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b5bb44845g5xgx"
May 25 11:47:47.507263 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:47:47.507267    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-qjvzz\" (UniqueName: \"kubernetes.io/projected/ce4cff5d-2132-4b5c-9aea-790191e87b00-kube-api-access-qjvzz\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b5bb44845g5xgx\" (UID: \"ce4cff5d-2132-4b5c-9aea-790191e87b00\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b5bb44845g5xgx"
May 25 11:47:47.507579 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:47:47.507311    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ce4cff5d-2132-4b5c-9aea-790191e87b00-home\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b5bb44845g5xgx\" (UID: \"ce4cff5d-2132-4b5c-9aea-790191e87b00\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b5bb44845g5xgx"
May 25 11:47:47.507579 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:47:47.507352    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce4cff5d-2132-4b5c-9aea-790191e87b00-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b5bb44845g5xgx\" (UID: \"ce4cff5d-2132-4b5c-9aea-790191e87b00\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b5bb44845g5xgx"
May 25 11:47:47.507579 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:47:47.507425    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce4cff5d-2132-4b5c-9aea-790191e87b00-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b5bb44845g5xgx\" (UID: \"ce4cff5d-2132-4b5c-9aea-790191e87b00\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b5bb44845g5xgx"
May 25 11:47:47.507579 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:47:47.507452    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ce4cff5d-2132-4b5c-9aea-790191e87b00-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b5bb44845g5xgx\" (UID: \"ce4cff5d-2132-4b5c-9aea-790191e87b00\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b5bb44845g5xgx"
May 25 11:47:47.507579 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:47:47.507472    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce4cff5d-2132-4b5c-9aea-790191e87b00-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b5bb44845g5xgx\" (UID: \"ce4cff5d-2132-4b5c-9aea-790191e87b00\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b5bb44845g5xgx"
May 25 11:47:47.507862 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:47:47.507784    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ce4cff5d-2132-4b5c-9aea-790191e87b00-home\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b5bb44845g5xgx\" (UID: \"ce4cff5d-2132-4b5c-9aea-790191e87b00\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b5bb44845g5xgx"
May 25 11:47:47.507862 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:47:47.507831    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce4cff5d-2132-4b5c-9aea-790191e87b00-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b5bb44845g5xgx\" (UID: \"ce4cff5d-2132-4b5c-9aea-790191e87b00\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b5bb44845g5xgx"
May 25 11:47:47.507966 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:47:47.507915    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ce4cff5d-2132-4b5c-9aea-790191e87b00-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b5bb44845g5xgx\" (UID: \"ce4cff5d-2132-4b5c-9aea-790191e87b00\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b5bb44845g5xgx"
May 25 11:47:47.508210 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:47:47.508183    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce4cff5d-2132-4b5c-9aea-790191e87b00-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b5bb44845g5xgx\" (UID: \"ce4cff5d-2132-4b5c-9aea-790191e87b00\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b5bb44845g5xgx"
May 25 11:47:47.509642 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:47:47.509622    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce4cff5d-2132-4b5c-9aea-790191e87b00-dshm\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b5bb44845g5xgx\" (UID: \"ce4cff5d-2132-4b5c-9aea-790191e87b00\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b5bb44845g5xgx"
May 25 11:47:47.510024 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:47:47.510003    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce4cff5d-2132-4b5c-9aea-790191e87b00-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b5bb44845g5xgx\" (UID: \"ce4cff5d-2132-4b5c-9aea-790191e87b00\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b5bb44845g5xgx"
May 25 11:47:47.516044 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:47:47.516021    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjvzz\" (UniqueName: \"kubernetes.io/projected/ce4cff5d-2132-4b5c-9aea-790191e87b00-kube-api-access-qjvzz\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b5bb44845g5xgx\" (UID: \"ce4cff5d-2132-4b5c-9aea-790191e87b00\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b5bb44845g5xgx"
May 25 11:47:47.617095 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:47:47.617014    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b5bb44845g5xgx"
May 25 11:47:47.769247 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:47:47.769224    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b5bb44845g5xgx"]
May 25 11:47:48.414155 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:47:48.414115    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b5bb44845g5xgx" event={"ID":"ce4cff5d-2132-4b5c-9aea-790191e87b00","Type":"ContainerStarted","Data":"9d64f4e18ba53c5fd64d0a1a9f0ccce86b55fdd178d4169b2320620b3db4d8da"}
May 25 11:47:48.414155 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:47:48.414154    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b5bb44845g5xgx" event={"ID":"ce4cff5d-2132-4b5c-9aea-790191e87b00","Type":"ContainerStarted","Data":"8bd2b9f0215d5d9d063c6f25d3c9f6c95e6bdc9213a5ec846211ad9ac422a248"}
May 25 11:47:49.187112 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:47:49.187072    2576 cadvisor_stats_provider.go:525] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad589270_6564_4cdd_a936_e645856ca27c.slice/crio-conmon-1135587c9ed5c0967dff249c1977243ae12238214bca4e11a83de03f01e28f00.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad589270_6564_4cdd_a936_e645856ca27c.slice/crio-1135587c9ed5c0967dff249c1977243ae12238214bca4e11a83de03f01e28f00.scope\": RecentStats: unable to find data in memory cache]"
May 25 11:47:49.338841 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:47:49.338818    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_conv-test-criticality-kserve-7c79ff4697-g6zqz_ad589270-6564-4cdd-a936-e645856ca27c/main/0.log"
May 25 11:47:49.339344 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:47:49.339321    2576 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/conv-test-criticality-kserve-7c79ff4697-g6zqz"
May 25 11:47:49.418388 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:47:49.418356    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_conv-test-criticality-kserve-7c79ff4697-g6zqz_ad589270-6564-4cdd-a936-e645856ca27c/main/0.log"
May 25 11:47:49.418795 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:47:49.418710    2576 generic.go:358] "Generic (PLEG): container finished" podID="ad589270-6564-4cdd-a936-e645856ca27c" containerID="1135587c9ed5c0967dff249c1977243ae12238214bca4e11a83de03f01e28f00" exitCode=137
May 25 11:47:49.418795 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:47:49.418787    2576 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/conv-test-criticality-kserve-7c79ff4697-g6zqz"
May 25 11:47:49.418874 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:47:49.418787    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/conv-test-criticality-kserve-7c79ff4697-g6zqz" event={"ID":"ad589270-6564-4cdd-a936-e645856ca27c","Type":"ContainerDied","Data":"1135587c9ed5c0967dff249c1977243ae12238214bca4e11a83de03f01e28f00"}
May 25 11:47:49.418874 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:47:49.418826    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/conv-test-criticality-kserve-7c79ff4697-g6zqz" event={"ID":"ad589270-6564-4cdd-a936-e645856ca27c","Type":"ContainerDied","Data":"8a6238c47367a8c513a1d2ba8daf0b0b81f486a18f08af7a4656370d706bc18c"}
May 25 11:47:49.418874 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:47:49.418842    2576 scope.go:117] "RemoveContainer" containerID="1135587c9ed5c0967dff249c1977243ae12238214bca4e11a83de03f01e28f00"
May 25 11:47:49.425685 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:47:49.425663    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad589270-6564-4cdd-a936-e645856ca27c-tls-certs\") pod \"ad589270-6564-4cdd-a936-e645856ca27c\" (UID: \"ad589270-6564-4cdd-a936-e645856ca27c\") "
May 25 11:47:49.425806 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:47:49.425724    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ad589270-6564-4cdd-a936-e645856ca27c-kserve-provision-location\") pod \"ad589270-6564-4cdd-a936-e645856ca27c\" (UID: \"ad589270-6564-4cdd-a936-e645856ca27c\") "
May 25 11:47:49.425806 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:47:49.425763    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-678tk\" (UniqueName: \"kubernetes.io/projected/ad589270-6564-4cdd-a936-e645856ca27c-kube-api-access-678tk\") pod \"ad589270-6564-4cdd-a936-e645856ca27c\" (UID: \"ad589270-6564-4cdd-a936-e645856ca27c\") "
May 25 11:47:49.425999 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:47:49.425831    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad589270-6564-4cdd-a936-e645856ca27c-dshm\") pod \"ad589270-6564-4cdd-a936-e645856ca27c\" (UID: \"ad589270-6564-4cdd-a936-e645856ca27c\") "
May 25 11:47:49.425999 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:47:49.425863    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad589270-6564-4cdd-a936-e645856ca27c-tmp-dir\") pod \"ad589270-6564-4cdd-a936-e645856ca27c\" (UID: \"ad589270-6564-4cdd-a936-e645856ca27c\") "
May 25 11:47:49.425999 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:47:49.425878    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad589270-6564-4cdd-a936-e645856ca27c-model-cache\") pod \"ad589270-6564-4cdd-a936-e645856ca27c\" (UID: \"ad589270-6564-4cdd-a936-e645856ca27c\") "
May 25 11:47:49.425999 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:47:49.425898    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad589270-6564-4cdd-a936-e645856ca27c-home\") pod \"ad589270-6564-4cdd-a936-e645856ca27c\" (UID: \"ad589270-6564-4cdd-a936-e645856ca27c\") "
May 25 11:47:49.426348 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:47:49.426237    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad589270-6564-4cdd-a936-e645856ca27c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ad589270-6564-4cdd-a936-e645856ca27c" (UID: "ad589270-6564-4cdd-a936-e645856ca27c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:47:49.426540 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:47:49.426511    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad589270-6564-4cdd-a936-e645856ca27c-model-cache" (OuterVolumeSpecName: "model-cache") pod "ad589270-6564-4cdd-a936-e645856ca27c" (UID: "ad589270-6564-4cdd-a936-e645856ca27c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:47:49.426622 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:47:49.426580    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad589270-6564-4cdd-a936-e645856ca27c-home" (OuterVolumeSpecName: "home") pod "ad589270-6564-4cdd-a936-e645856ca27c" (UID: "ad589270-6564-4cdd-a936-e645856ca27c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:47:49.428089 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:47:49.428068    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad589270-6564-4cdd-a936-e645856ca27c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ad589270-6564-4cdd-a936-e645856ca27c" (UID: "ad589270-6564-4cdd-a936-e645856ca27c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 11:47:49.428610 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:47:49.428592    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad589270-6564-4cdd-a936-e645856ca27c-kube-api-access-678tk" (OuterVolumeSpecName: "kube-api-access-678tk") pod "ad589270-6564-4cdd-a936-e645856ca27c" (UID: "ad589270-6564-4cdd-a936-e645856ca27c"). InnerVolumeSpecName "kube-api-access-678tk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 11:47:49.428846 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:47:49.428825    2576 scope.go:117] "RemoveContainer" containerID="357a749bc4aacab371ac072280e5589af1871a2a271b33cf6331ec4c125f8c23"
May 25 11:47:49.429175 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:47:49.429149    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad589270-6564-4cdd-a936-e645856ca27c-dshm" (OuterVolumeSpecName: "dshm") pod "ad589270-6564-4cdd-a936-e645856ca27c" (UID: "ad589270-6564-4cdd-a936-e645856ca27c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:47:49.496937 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:47:49.496885    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad589270-6564-4cdd-a936-e645856ca27c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ad589270-6564-4cdd-a936-e645856ca27c" (UID: "ad589270-6564-4cdd-a936-e645856ca27c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:47:49.505087 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:47:49.505068    2576 scope.go:117] "RemoveContainer" containerID="1135587c9ed5c0967dff249c1977243ae12238214bca4e11a83de03f01e28f00"
May 25 11:47:49.505447 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:47:49.505413    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1135587c9ed5c0967dff249c1977243ae12238214bca4e11a83de03f01e28f00\": container with ID starting with 1135587c9ed5c0967dff249c1977243ae12238214bca4e11a83de03f01e28f00 not found: ID does not exist" containerID="1135587c9ed5c0967dff249c1977243ae12238214bca4e11a83de03f01e28f00"
May 25 11:47:49.505545 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:47:49.505455    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1135587c9ed5c0967dff249c1977243ae12238214bca4e11a83de03f01e28f00"} err="failed to get container status \"1135587c9ed5c0967dff249c1977243ae12238214bca4e11a83de03f01e28f00\": rpc error: code = NotFound desc = could not find container \"1135587c9ed5c0967dff249c1977243ae12238214bca4e11a83de03f01e28f00\": container with ID starting with 1135587c9ed5c0967dff249c1977243ae12238214bca4e11a83de03f01e28f00 not found: ID does not exist"
May 25 11:47:49.505545 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:47:49.505485    2576 scope.go:117] "RemoveContainer" containerID="357a749bc4aacab371ac072280e5589af1871a2a271b33cf6331ec4c125f8c23"
May 25 11:47:49.505754 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:47:49.505734    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"357a749bc4aacab371ac072280e5589af1871a2a271b33cf6331ec4c125f8c23\": container with ID starting with 357a749bc4aacab371ac072280e5589af1871a2a271b33cf6331ec4c125f8c23 not found: ID does not exist" containerID="357a749bc4aacab371ac072280e5589af1871a2a271b33cf6331ec4c125f8c23"
May 25 11:47:49.505821 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:47:49.505759    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"357a749bc4aacab371ac072280e5589af1871a2a271b33cf6331ec4c125f8c23"} err="failed to get container status \"357a749bc4aacab371ac072280e5589af1871a2a271b33cf6331ec4c125f8c23\": rpc error: code = NotFound desc = could not find container \"357a749bc4aacab371ac072280e5589af1871a2a271b33cf6331ec4c125f8c23\": container with ID starting with 357a749bc4aacab371ac072280e5589af1871a2a271b33cf6331ec4c125f8c23 not found: ID does not exist"
May 25 11:47:49.526995 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:47:49.526969    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad589270-6564-4cdd-a936-e645856ca27c-dshm\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
May 25 11:47:49.526995 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:47:49.526994    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad589270-6564-4cdd-a936-e645856ca27c-tmp-dir\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
May 25 11:47:49.527149 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:47:49.527004    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad589270-6564-4cdd-a936-e645856ca27c-model-cache\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
May 25 11:47:49.527149 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:47:49.527012    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad589270-6564-4cdd-a936-e645856ca27c-home\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
May 25 11:47:49.527149 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:47:49.527023    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad589270-6564-4cdd-a936-e645856ca27c-tls-certs\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
May 25 11:47:49.527149 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:47:49.527037    2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ad589270-6564-4cdd-a936-e645856ca27c-kserve-provision-location\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
May 25 11:47:49.527149 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:47:49.527050    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-678tk\" (UniqueName: \"kubernetes.io/projected/ad589270-6564-4cdd-a936-e645856ca27c-kube-api-access-678tk\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
May 25 11:47:49.741738 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:47:49.741712    2576 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/conv-test-criticality-kserve-7c79ff4697-g6zqz"]
May 25 11:47:49.747989 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:47:49.747962    2576 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/conv-test-criticality-kserve-7c79ff4697-g6zqz"]
May 25 11:47:51.098707 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:47:51.098676    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ad589270-6564-4cdd-a936-e645856ca27c" path="/var/lib/kubelet/pods/ad589270-6564-4cdd-a936-e645856ca27c/volumes"
May 25 11:47:54.437509 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:47:54.437480    2576 generic.go:358] "Generic (PLEG): container finished" podID="ce4cff5d-2132-4b5c-9aea-790191e87b00" containerID="9d64f4e18ba53c5fd64d0a1a9f0ccce86b55fdd178d4169b2320620b3db4d8da" exitCode=0
May 25 11:47:54.437848 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:47:54.437547    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b5bb44845g5xgx" event={"ID":"ce4cff5d-2132-4b5c-9aea-790191e87b00","Type":"ContainerDied","Data":"9d64f4e18ba53c5fd64d0a1a9f0ccce86b55fdd178d4169b2320620b3db4d8da"}
May 25 11:47:55.442704 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:47:55.442673    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b5bb44845g5xgx" event={"ID":"ce4cff5d-2132-4b5c-9aea-790191e87b00","Type":"ContainerStarted","Data":"71c7ed88b79de20a8167e3bc3e0f2c7a790f58d2bdc3b608685f12a5a710efc6"}
May 25 11:47:55.463592 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:47:55.463545    2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b5bb44845g5xgx" podStartSLOduration=8.463530459 podStartE2EDuration="8.463530459s" podCreationTimestamp="2026-05-25 11:47:47 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 11:47:55.462685979 +0000 UTC m=+1820.980417436" watchObservedRunningTime="2026-05-25 11:47:55.463530459 +0000 UTC m=+1820.981261942"
May 25 11:47:57.617504 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:47:57.617462    2576 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b5bb44845g5xgx"
May 25 11:47:57.617504 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:47:57.617508    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b5bb44845g5xgx"
May 25 11:47:57.618978 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:47:57.618946    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b5bb44845g5xgx" podUID="ce4cff5d-2132-4b5c-9aea-790191e87b00" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/health\": dial tcp 10.133.0.42:8000: connect: connection refused"
May 25 11:48:07.618300 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:48:07.618234    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b5bb44845g5xgx" podUID="ce4cff5d-2132-4b5c-9aea-790191e87b00" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/health\": dial tcp 10.133.0.42:8000: connect: connection refused"
May 25 11:48:10.636112 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:48:10.636060    2576 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-799dc8b8c-8w6n5" podUID="e96af524-82a6-41ef-8bf2-8a1bf59bc33f" containerName="llm-d-routing-sidecar" containerID="cri-o://ccd7f1468d92f06063bfdd6d854e68e1b9b7d93b085f4b90887f8eddc795a286" gracePeriod=2
May 25 11:48:10.947806 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:48:10.947786    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-799dc8b8c-8w6n5_e96af524-82a6-41ef-8bf2-8a1bf59bc33f/main/0.log"
May 25 11:48:10.948529 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:48:10.948508    2576 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-799dc8b8c-8w6n5"
May 25 11:48:10.951788 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:48:10.951772    2576 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-78459cb79d-92lfb"
May 25 11:48:11.127953 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:48:11.127871    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/887d4410-aee9-44bc-99ee-9fa600b7b6ac-tls-certs\") pod \"887d4410-aee9-44bc-99ee-9fa600b7b6ac\" (UID: \"887d4410-aee9-44bc-99ee-9fa600b7b6ac\") "
May 25 11:48:11.127953 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:48:11.127929    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e96af524-82a6-41ef-8bf2-8a1bf59bc33f-tmp-dir\") pod \"e96af524-82a6-41ef-8bf2-8a1bf59bc33f\" (UID: \"e96af524-82a6-41ef-8bf2-8a1bf59bc33f\") "
May 25 11:48:11.128163 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:48:11.127962    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h824v\" (UniqueName: \"kubernetes.io/projected/e96af524-82a6-41ef-8bf2-8a1bf59bc33f-kube-api-access-h824v\") pod \"e96af524-82a6-41ef-8bf2-8a1bf59bc33f\" (UID: \"e96af524-82a6-41ef-8bf2-8a1bf59bc33f\") "
May 25 11:48:11.128163 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:48:11.127990    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/887d4410-aee9-44bc-99ee-9fa600b7b6ac-home\") pod \"887d4410-aee9-44bc-99ee-9fa600b7b6ac\" (UID: \"887d4410-aee9-44bc-99ee-9fa600b7b6ac\") "
May 25 11:48:11.128163 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:48:11.128017    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e96af524-82a6-41ef-8bf2-8a1bf59bc33f-kserve-provision-location\") pod \"e96af524-82a6-41ef-8bf2-8a1bf59bc33f\" (UID: \"e96af524-82a6-41ef-8bf2-8a1bf59bc33f\") "
May 25 11:48:11.128163 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:48:11.128062    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/887d4410-aee9-44bc-99ee-9fa600b7b6ac-model-cache\") pod \"887d4410-aee9-44bc-99ee-9fa600b7b6ac\" (UID: \"887d4410-aee9-44bc-99ee-9fa600b7b6ac\") "
May 25 11:48:11.128163 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:48:11.128089    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e96af524-82a6-41ef-8bf2-8a1bf59bc33f-home\") pod \"e96af524-82a6-41ef-8bf2-8a1bf59bc33f\" (UID: \"e96af524-82a6-41ef-8bf2-8a1bf59bc33f\") "
May 25 11:48:11.128163 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:48:11.128111    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e96af524-82a6-41ef-8bf2-8a1bf59bc33f-tls-certs\") pod \"e96af524-82a6-41ef-8bf2-8a1bf59bc33f\" (UID: \"e96af524-82a6-41ef-8bf2-8a1bf59bc33f\") "
May 25 11:48:11.128163 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:48:11.128154    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/887d4410-aee9-44bc-99ee-9fa600b7b6ac-dshm\") pod \"887d4410-aee9-44bc-99ee-9fa600b7b6ac\" (UID: \"887d4410-aee9-44bc-99ee-9fa600b7b6ac\") "
May 25 11:48:11.128545 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:48:11.128250    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/887d4410-aee9-44bc-99ee-9fa600b7b6ac-tmp-dir\") pod \"887d4410-aee9-44bc-99ee-9fa600b7b6ac\" (UID: \"887d4410-aee9-44bc-99ee-9fa600b7b6ac\") "
May 25 11:48:11.128545 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:48:11.128295    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e96af524-82a6-41ef-8bf2-8a1bf59bc33f-dshm\") pod \"e96af524-82a6-41ef-8bf2-8a1bf59bc33f\" (UID: \"e96af524-82a6-41ef-8bf2-8a1bf59bc33f\") "
May 25 11:48:11.128545 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:48:11.128336    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w6qmg\" (UniqueName: \"kubernetes.io/projected/887d4410-aee9-44bc-99ee-9fa600b7b6ac-kube-api-access-w6qmg\") pod \"887d4410-aee9-44bc-99ee-9fa600b7b6ac\" (UID: \"887d4410-aee9-44bc-99ee-9fa600b7b6ac\") "
May 25 11:48:11.128545 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:48:11.128361    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e96af524-82a6-41ef-8bf2-8a1bf59bc33f-model-cache\") pod \"e96af524-82a6-41ef-8bf2-8a1bf59bc33f\" (UID: \"e96af524-82a6-41ef-8bf2-8a1bf59bc33f\") "
May 25 11:48:11.128545 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:48:11.128425    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/887d4410-aee9-44bc-99ee-9fa600b7b6ac-kserve-provision-location\") pod \"887d4410-aee9-44bc-99ee-9fa600b7b6ac\" (UID: \"887d4410-aee9-44bc-99ee-9fa600b7b6ac\") "
May 25 11:48:11.128788 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:48:11.128715    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/887d4410-aee9-44bc-99ee-9fa600b7b6ac-home" (OuterVolumeSpecName: "home") pod "887d4410-aee9-44bc-99ee-9fa600b7b6ac" (UID: "887d4410-aee9-44bc-99ee-9fa600b7b6ac"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:48:11.130221 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:48:11.130190    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/887d4410-aee9-44bc-99ee-9fa600b7b6ac-model-cache" (OuterVolumeSpecName: "model-cache") pod "887d4410-aee9-44bc-99ee-9fa600b7b6ac" (UID: "887d4410-aee9-44bc-99ee-9fa600b7b6ac"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:48:11.130766 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:48:11.130537    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/887d4410-aee9-44bc-99ee-9fa600b7b6ac-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "887d4410-aee9-44bc-99ee-9fa600b7b6ac" (UID: "887d4410-aee9-44bc-99ee-9fa600b7b6ac"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 11:48:11.130766 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:48:11.130655    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e96af524-82a6-41ef-8bf2-8a1bf59bc33f-kube-api-access-h824v" (OuterVolumeSpecName: "kube-api-access-h824v") pod "e96af524-82a6-41ef-8bf2-8a1bf59bc33f" (UID: "e96af524-82a6-41ef-8bf2-8a1bf59bc33f"). InnerVolumeSpecName "kube-api-access-h824v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 11:48:11.131378 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:48:11.131356    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e96af524-82a6-41ef-8bf2-8a1bf59bc33f-home" (OuterVolumeSpecName: "home") pod "e96af524-82a6-41ef-8bf2-8a1bf59bc33f" (UID: "e96af524-82a6-41ef-8bf2-8a1bf59bc33f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:48:11.132476 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:48:11.131550    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e96af524-82a6-41ef-8bf2-8a1bf59bc33f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e96af524-82a6-41ef-8bf2-8a1bf59bc33f" (UID: "e96af524-82a6-41ef-8bf2-8a1bf59bc33f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 11:48:11.132476 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:48:11.131771    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e96af524-82a6-41ef-8bf2-8a1bf59bc33f-model-cache" (OuterVolumeSpecName: "model-cache") pod "e96af524-82a6-41ef-8bf2-8a1bf59bc33f" (UID: "e96af524-82a6-41ef-8bf2-8a1bf59bc33f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:48:11.132850 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:48:11.132824    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/887d4410-aee9-44bc-99ee-9fa600b7b6ac-kube-api-access-w6qmg" (OuterVolumeSpecName: "kube-api-access-w6qmg") pod "887d4410-aee9-44bc-99ee-9fa600b7b6ac" (UID: "887d4410-aee9-44bc-99ee-9fa600b7b6ac"). InnerVolumeSpecName "kube-api-access-w6qmg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 11:48:11.133158 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:48:11.133129    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e96af524-82a6-41ef-8bf2-8a1bf59bc33f-dshm" (OuterVolumeSpecName: "dshm") pod "e96af524-82a6-41ef-8bf2-8a1bf59bc33f" (UID: "e96af524-82a6-41ef-8bf2-8a1bf59bc33f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:48:11.133844 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:48:11.133822    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/887d4410-aee9-44bc-99ee-9fa600b7b6ac-dshm" (OuterVolumeSpecName: "dshm") pod "887d4410-aee9-44bc-99ee-9fa600b7b6ac" (UID: "887d4410-aee9-44bc-99ee-9fa600b7b6ac"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:48:11.141241 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:48:11.141214    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e96af524-82a6-41ef-8bf2-8a1bf59bc33f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e96af524-82a6-41ef-8bf2-8a1bf59bc33f" (UID: "e96af524-82a6-41ef-8bf2-8a1bf59bc33f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:48:11.148994 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:48:11.148969    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/887d4410-aee9-44bc-99ee-9fa600b7b6ac-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "887d4410-aee9-44bc-99ee-9fa600b7b6ac" (UID: "887d4410-aee9-44bc-99ee-9fa600b7b6ac"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:48:11.192878 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:48:11.192814    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/887d4410-aee9-44bc-99ee-9fa600b7b6ac-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "887d4410-aee9-44bc-99ee-9fa600b7b6ac" (UID: "887d4410-aee9-44bc-99ee-9fa600b7b6ac"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:48:11.197368 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:48:11.197341    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e96af524-82a6-41ef-8bf2-8a1bf59bc33f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e96af524-82a6-41ef-8bf2-8a1bf59bc33f" (UID: "e96af524-82a6-41ef-8bf2-8a1bf59bc33f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:48:11.229562 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:48:11.229493    2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/887d4410-aee9-44bc-99ee-9fa600b7b6ac-kserve-provision-location\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
May 25 11:48:11.229562 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:48:11.229562    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/887d4410-aee9-44bc-99ee-9fa600b7b6ac-tls-certs\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
May 25 11:48:11.229759 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:48:11.229580    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e96af524-82a6-41ef-8bf2-8a1bf59bc33f-tmp-dir\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
May 25 11:48:11.229759 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:48:11.229596    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h824v\" (UniqueName: \"kubernetes.io/projected/e96af524-82a6-41ef-8bf2-8a1bf59bc33f-kube-api-access-h824v\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
May 25 11:48:11.229759 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:48:11.229612    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/887d4410-aee9-44bc-99ee-9fa600b7b6ac-home\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
May 25 11:48:11.229759 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:48:11.229627    2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e96af524-82a6-41ef-8bf2-8a1bf59bc33f-kserve-provision-location\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
May 25 11:48:11.229759 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:48:11.229642    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/887d4410-aee9-44bc-99ee-9fa600b7b6ac-model-cache\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
May 25 11:48:11.229759 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:48:11.229655    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e96af524-82a6-41ef-8bf2-8a1bf59bc33f-home\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
May 25 11:48:11.229759 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:48:11.229668    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e96af524-82a6-41ef-8bf2-8a1bf59bc33f-tls-certs\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
May 25 11:48:11.229759 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:48:11.229680    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/887d4410-aee9-44bc-99ee-9fa600b7b6ac-dshm\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
May 25 11:48:11.229759 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:48:11.229692    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/887d4410-aee9-44bc-99ee-9fa600b7b6ac-tmp-dir\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
May 25 11:48:11.229759 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:48:11.229704    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e96af524-82a6-41ef-8bf2-8a1bf59bc33f-dshm\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
May 25 11:48:11.229759 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:48:11.229717    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w6qmg\" (UniqueName: \"kubernetes.io/projected/887d4410-aee9-44bc-99ee-9fa600b7b6ac-kube-api-access-w6qmg\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
May 25 11:48:11.229759 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:48:11.229730    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e96af524-82a6-41ef-8bf2-8a1bf59bc33f-model-cache\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
May 25 11:48:11.497734 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:48:11.497703    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-799dc8b8c-8w6n5_e96af524-82a6-41ef-8bf2-8a1bf59bc33f/main/0.log"
May 25 11:48:11.498414 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:48:11.498383    2576 generic.go:358] "Generic (PLEG): container finished" podID="e96af524-82a6-41ef-8bf2-8a1bf59bc33f" containerID="b60aa49e873e79bbea2ce5a73b8123ed1f3fcac0abc08b5393ed0aaf61aa767d" exitCode=137
May 25 11:48:11.498414 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:48:11.498412    2576 generic.go:358] "Generic (PLEG): container finished" podID="e96af524-82a6-41ef-8bf2-8a1bf59bc33f" containerID="ccd7f1468d92f06063bfdd6d854e68e1b9b7d93b085f4b90887f8eddc795a286" exitCode=0
May 25 11:48:11.498618 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:48:11.498472    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-799dc8b8c-8w6n5" event={"ID":"e96af524-82a6-41ef-8bf2-8a1bf59bc33f","Type":"ContainerDied","Data":"b60aa49e873e79bbea2ce5a73b8123ed1f3fcac0abc08b5393ed0aaf61aa767d"}
May 25 11:48:11.498618 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:48:11.498501    2576 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-799dc8b8c-8w6n5"
May 25 11:48:11.498618 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:48:11.498520    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-799dc8b8c-8w6n5" event={"ID":"e96af524-82a6-41ef-8bf2-8a1bf59bc33f","Type":"ContainerDied","Data":"ccd7f1468d92f06063bfdd6d854e68e1b9b7d93b085f4b90887f8eddc795a286"}
May 25 11:48:11.498618 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:48:11.498532    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-799dc8b8c-8w6n5" event={"ID":"e96af524-82a6-41ef-8bf2-8a1bf59bc33f","Type":"ContainerDied","Data":"aae05d56c6e7bfe3d01ab60827ea6f64dd9bff60c5918d6eab8862266a053e48"}
May 25 11:48:11.498618 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:48:11.498546    2576 scope.go:117] "RemoveContainer" containerID="b60aa49e873e79bbea2ce5a73b8123ed1f3fcac0abc08b5393ed0aaf61aa767d"
May 25 11:48:11.500151 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:48:11.500124    2576 generic.go:358] "Generic (PLEG): container finished" podID="887d4410-aee9-44bc-99ee-9fa600b7b6ac" containerID="bd4f18fb0669d2bc012182ce6c58348fb7fdd977ce0815d73a782ca0e48bc728" exitCode=137
May 25 11:48:11.500269 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:48:11.500161    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-78459cb79d-92lfb" event={"ID":"887d4410-aee9-44bc-99ee-9fa600b7b6ac","Type":"ContainerDied","Data":"bd4f18fb0669d2bc012182ce6c58348fb7fdd977ce0815d73a782ca0e48bc728"}
May 25 11:48:11.500269 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:48:11.500190    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-78459cb79d-92lfb" event={"ID":"887d4410-aee9-44bc-99ee-9fa600b7b6ac","Type":"ContainerDied","Data":"a47be0c9e000bc6f5b13ae325bda2c7bcade6c8322e8aa43f6b1fe42558b768d"}
May 25 11:48:11.500269 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:48:11.500249    2576 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-78459cb79d-92lfb"
May 25 11:48:11.508478 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:48:11.508459    2576 scope.go:117] "RemoveContainer" containerID="e1a0f315b50db54160ac4a8e7ce07441060df2a3c37274f4ecfbee9a15763540"
May 25 11:48:11.519983 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:48:11.519961    2576 scope.go:117] "RemoveContainer" containerID="ccd7f1468d92f06063bfdd6d854e68e1b9b7d93b085f4b90887f8eddc795a286"
May 25 11:48:11.524271 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:48:11.524247    2576 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-799dc8b8c-8w6n5"]
May 25 11:48:11.527548 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:48:11.527524    2576 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-799dc8b8c-8w6n5"]
May 25 11:48:11.530095 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:48:11.530062    2576 scope.go:117] "RemoveContainer" containerID="b60aa49e873e79bbea2ce5a73b8123ed1f3fcac0abc08b5393ed0aaf61aa767d"
May 25 11:48:11.530427 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:48:11.530407    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b60aa49e873e79bbea2ce5a73b8123ed1f3fcac0abc08b5393ed0aaf61aa767d\": container with ID starting with b60aa49e873e79bbea2ce5a73b8123ed1f3fcac0abc08b5393ed0aaf61aa767d not found: ID does not exist" containerID="b60aa49e873e79bbea2ce5a73b8123ed1f3fcac0abc08b5393ed0aaf61aa767d"
May 25 11:48:11.530514 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:48:11.530434    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b60aa49e873e79bbea2ce5a73b8123ed1f3fcac0abc08b5393ed0aaf61aa767d"} err="failed to get container status \"b60aa49e873e79bbea2ce5a73b8123ed1f3fcac0abc08b5393ed0aaf61aa767d\": rpc error: code = NotFound desc = could not find container \"b60aa49e873e79bbea2ce5a73b8123ed1f3fcac0abc08b5393ed0aaf61aa767d\": container with ID starting with b60aa49e873e79bbea2ce5a73b8123ed1f3fcac0abc08b5393ed0aaf61aa767d not found: ID does not exist"
May 25 11:48:11.530514 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:48:11.530452    2576 scope.go:117] "RemoveContainer" containerID="e1a0f315b50db54160ac4a8e7ce07441060df2a3c37274f4ecfbee9a15763540"
May 25 11:48:11.530719 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:48:11.530701    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1a0f315b50db54160ac4a8e7ce07441060df2a3c37274f4ecfbee9a15763540\": container with ID starting with e1a0f315b50db54160ac4a8e7ce07441060df2a3c37274f4ecfbee9a15763540 not found: ID does not exist" containerID="e1a0f315b50db54160ac4a8e7ce07441060df2a3c37274f4ecfbee9a15763540"
May 25 11:48:11.530783 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:48:11.530725    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1a0f315b50db54160ac4a8e7ce07441060df2a3c37274f4ecfbee9a15763540"} err="failed to get container status \"e1a0f315b50db54160ac4a8e7ce07441060df2a3c37274f4ecfbee9a15763540\": rpc error: code = NotFound desc = could not find container \"e1a0f315b50db54160ac4a8e7ce07441060df2a3c37274f4ecfbee9a15763540\": container with ID starting with e1a0f315b50db54160ac4a8e7ce07441060df2a3c37274f4ecfbee9a15763540 not found: ID does not exist"
May 25 11:48:11.530783 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:48:11.530741    2576 scope.go:117] "RemoveContainer" containerID="ccd7f1468d92f06063bfdd6d854e68e1b9b7d93b085f4b90887f8eddc795a286"
May 25 11:48:11.530980 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:48:11.530966    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ccd7f1468d92f06063bfdd6d854e68e1b9b7d93b085f4b90887f8eddc795a286\": container with ID starting with ccd7f1468d92f06063bfdd6d854e68e1b9b7d93b085f4b90887f8eddc795a286 not found: ID does not exist" containerID="ccd7f1468d92f06063bfdd6d854e68e1b9b7d93b085f4b90887f8eddc795a286"
May 25 11:48:11.531038 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:48:11.530983    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ccd7f1468d92f06063bfdd6d854e68e1b9b7d93b085f4b90887f8eddc795a286"} err="failed to get container status \"ccd7f1468d92f06063bfdd6d854e68e1b9b7d93b085f4b90887f8eddc795a286\": rpc error: code = NotFound desc = could not find container \"ccd7f1468d92f06063bfdd6d854e68e1b9b7d93b085f4b90887f8eddc795a286\": container with ID starting with ccd7f1468d92f06063bfdd6d854e68e1b9b7d93b085f4b90887f8eddc795a286 not found: ID does not exist"
May 25 11:48:11.531038 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:48:11.531003    2576 scope.go:117] "RemoveContainer" containerID="b60aa49e873e79bbea2ce5a73b8123ed1f3fcac0abc08b5393ed0aaf61aa767d"
May 25 11:48:11.531214 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:48:11.531196    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b60aa49e873e79bbea2ce5a73b8123ed1f3fcac0abc08b5393ed0aaf61aa767d"} err="failed to get container status \"b60aa49e873e79bbea2ce5a73b8123ed1f3fcac0abc08b5393ed0aaf61aa767d\": rpc error: code = NotFound desc = could not find container \"b60aa49e873e79bbea2ce5a73b8123ed1f3fcac0abc08b5393ed0aaf61aa767d\": container with ID starting with b60aa49e873e79bbea2ce5a73b8123ed1f3fcac0abc08b5393ed0aaf61aa767d not found: ID does not exist"
May 25 11:48:11.531316 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:48:11.531215    2576 scope.go:117] "RemoveContainer" containerID="e1a0f315b50db54160ac4a8e7ce07441060df2a3c37274f4ecfbee9a15763540"
May 25 11:48:11.531457 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:48:11.531426    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1a0f315b50db54160ac4a8e7ce07441060df2a3c37274f4ecfbee9a15763540"} err="failed to get container status \"e1a0f315b50db54160ac4a8e7ce07441060df2a3c37274f4ecfbee9a15763540\": rpc error: code = NotFound desc = could not find container \"e1a0f315b50db54160ac4a8e7ce07441060df2a3c37274f4ecfbee9a15763540\": container with ID starting with e1a0f315b50db54160ac4a8e7ce07441060df2a3c37274f4ecfbee9a15763540 not found: ID does not exist"
May 25 11:48:11.531559 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:48:11.531458    2576 scope.go:117] "RemoveContainer" containerID="ccd7f1468d92f06063bfdd6d854e68e1b9b7d93b085f4b90887f8eddc795a286"
May 25 11:48:11.531727 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:48:11.531704    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ccd7f1468d92f06063bfdd6d854e68e1b9b7d93b085f4b90887f8eddc795a286"} err="failed to get container status \"ccd7f1468d92f06063bfdd6d854e68e1b9b7d93b085f4b90887f8eddc795a286\": rpc error: code = NotFound desc = could not find container \"ccd7f1468d92f06063bfdd6d854e68e1b9b7d93b085f4b90887f8eddc795a286\": container with ID starting with ccd7f1468d92f06063bfdd6d854e68e1b9b7d93b085f4b90887f8eddc795a286 not found: ID does not exist"
May 25 11:48:11.531783 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:48:11.531731    2576 scope.go:117] "RemoveContainer" containerID="bd4f18fb0669d2bc012182ce6c58348fb7fdd977ce0815d73a782ca0e48bc728"
May 25 11:48:11.539429 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:48:11.539401    2576 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-78459cb79d-92lfb"]
May 25 11:48:11.540719 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:48:11.540702    2576 scope.go:117] "RemoveContainer" containerID="2d12a14fe595187f73cf5b17df0af7f59862b696ca2b1d9e5a5fb797cbc9eb93"
May 25 11:48:11.546396 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:48:11.546372    2576 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-78459cb79d-92lfb"]
May 25 11:48:11.607425 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:48:11.607399    2576 scope.go:117] "RemoveContainer" containerID="bd4f18fb0669d2bc012182ce6c58348fb7fdd977ce0815d73a782ca0e48bc728"
May 25 11:48:11.607770 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:48:11.607751    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd4f18fb0669d2bc012182ce6c58348fb7fdd977ce0815d73a782ca0e48bc728\": container with ID starting with bd4f18fb0669d2bc012182ce6c58348fb7fdd977ce0815d73a782ca0e48bc728 not found: ID does not exist" containerID="bd4f18fb0669d2bc012182ce6c58348fb7fdd977ce0815d73a782ca0e48bc728"
May 25 11:48:11.607814 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:48:11.607779    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd4f18fb0669d2bc012182ce6c58348fb7fdd977ce0815d73a782ca0e48bc728"} err="failed to get container status \"bd4f18fb0669d2bc012182ce6c58348fb7fdd977ce0815d73a782ca0e48bc728\": rpc error: code = NotFound desc = could not find container \"bd4f18fb0669d2bc012182ce6c58348fb7fdd977ce0815d73a782ca0e48bc728\": container with ID starting with bd4f18fb0669d2bc012182ce6c58348fb7fdd977ce0815d73a782ca0e48bc728 not found: ID does not exist"
May 25 11:48:11.607814 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:48:11.607799    2576 scope.go:117] "RemoveContainer" containerID="2d12a14fe595187f73cf5b17df0af7f59862b696ca2b1d9e5a5fb797cbc9eb93"
May 25 11:48:11.608056 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:48:11.608036    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d12a14fe595187f73cf5b17df0af7f59862b696ca2b1d9e5a5fb797cbc9eb93\": container with ID starting with 2d12a14fe595187f73cf5b17df0af7f59862b696ca2b1d9e5a5fb797cbc9eb93 not found: ID does not exist" containerID="2d12a14fe595187f73cf5b17df0af7f59862b696ca2b1d9e5a5fb797cbc9eb93"
May 25 11:48:11.608111 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:48:11.608065    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d12a14fe595187f73cf5b17df0af7f59862b696ca2b1d9e5a5fb797cbc9eb93"} err="failed to get container status \"2d12a14fe595187f73cf5b17df0af7f59862b696ca2b1d9e5a5fb797cbc9eb93\": rpc error: code = NotFound desc = could not find container \"2d12a14fe595187f73cf5b17df0af7f59862b696ca2b1d9e5a5fb797cbc9eb93\": container with ID starting with 2d12a14fe595187f73cf5b17df0af7f59862b696ca2b1d9e5a5fb797cbc9eb93 not found: ID does not exist"
May 25 11:48:13.099875 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:48:13.099841    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="887d4410-aee9-44bc-99ee-9fa600b7b6ac" path="/var/lib/kubelet/pods/887d4410-aee9-44bc-99ee-9fa600b7b6ac/volumes"
May 25 11:48:13.100302 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:48:13.100268    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e96af524-82a6-41ef-8bf2-8a1bf59bc33f" path="/var/lib/kubelet/pods/e96af524-82a6-41ef-8bf2-8a1bf59bc33f/volumes"
May 25 11:48:17.617870 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:48:17.617823    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b5bb44845g5xgx" podUID="ce4cff5d-2132-4b5c-9aea-790191e87b00" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/health\": dial tcp 10.133.0.42:8000: connect: connection refused"
May 25 11:48:27.617895 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:48:27.617847    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b5bb44845g5xgx" podUID="ce4cff5d-2132-4b5c-9aea-790191e87b00" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/health\": dial tcp 10.133.0.42:8000: connect: connection refused"
May 25 11:48:30.571655 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:48:30.571618    2576 generic.go:358] "Generic (PLEG): container finished" podID="aacdbcbe-b12e-4969-9cfe-586811191878" containerID="040b762224fcfa7e740f1a88d058ff4f7a4ed8007612d1ddef1786617e90eb36" exitCode=0
May 25 11:48:30.572044 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:48:30.571683    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5449d77-tm4cr" event={"ID":"aacdbcbe-b12e-4969-9cfe-586811191878","Type":"ContainerDied","Data":"040b762224fcfa7e740f1a88d058ff4f7a4ed8007612d1ddef1786617e90eb36"}
May 25 11:48:30.572986 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:48:30.572966    2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 25 11:48:31.577911 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:48:31.577793    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5449d77-tm4cr" event={"ID":"aacdbcbe-b12e-4969-9cfe-586811191878","Type":"ContainerStarted","Data":"cc2b8e06e66a0a952f0a4d67be77249cfdbec0fac200d7b8c83dd5f5af544f48"}
May 25 11:48:31.609954 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:48:31.600196    2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5449d77-tm4cr" podStartSLOduration=74.600177892 podStartE2EDuration="1m14.600177892s" podCreationTimestamp="2026-05-25 11:47:17 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 11:48:31.596978462 +0000 UTC m=+1857.114709920" watchObservedRunningTime="2026-05-25 11:48:31.600177892 +0000 UTC m=+1857.117909349"
May 25 11:48:37.618201 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:48:37.618147    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b5bb44845g5xgx" podUID="ce4cff5d-2132-4b5c-9aea-790191e87b00" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/health\": dial tcp 10.133.0.42:8000: connect: connection refused"
May 25 11:48:37.692364 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:48:37.692332    2576 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5449d77-tm4cr"
May 25 11:48:37.692573 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:48:37.692375    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5449d77-tm4cr"
May 25 11:48:37.694106 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:48:37.694070    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5449d77-tm4cr" podUID="aacdbcbe-b12e-4969-9cfe-586811191878" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/health\": dial tcp 10.133.0.41:8000: connect: connection refused"
May 25 11:48:47.617651 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:48:47.617555    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b5bb44845g5xgx" podUID="ce4cff5d-2132-4b5c-9aea-790191e87b00" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/health\": dial tcp 10.133.0.42:8000: connect: connection refused"
May 25 11:48:47.693196 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:48:47.693147    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5449d77-tm4cr" podUID="aacdbcbe-b12e-4969-9cfe-586811191878" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/health\": dial tcp 10.133.0.41:8000: connect: connection refused"
May 25 11:48:57.617773 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:48:57.617724    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b5bb44845g5xgx" podUID="ce4cff5d-2132-4b5c-9aea-790191e87b00" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/health\": dial tcp 10.133.0.42:8000: connect: connection refused"
May 25 11:48:57.693386 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:48:57.693344    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5449d77-tm4cr" podUID="aacdbcbe-b12e-4969-9cfe-586811191878" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/health\": dial tcp 10.133.0.41:8000: connect: connection refused"
May 25 11:49:07.618337 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:49:07.618290    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b5bb44845g5xgx" podUID="ce4cff5d-2132-4b5c-9aea-790191e87b00" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/health\": dial tcp 10.133.0.42:8000: connect: connection refused"
May 25 11:49:07.692870 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:49:07.692818    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5449d77-tm4cr" podUID="aacdbcbe-b12e-4969-9cfe-586811191878" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/health\": dial tcp 10.133.0.41:8000: connect: connection refused"
May 25 11:49:17.617494 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:49:17.617437    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b5bb44845g5xgx" podUID="ce4cff5d-2132-4b5c-9aea-790191e87b00" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/health\": dial tcp 10.133.0.42:8000: connect: connection refused"
May 25 11:49:17.693382 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:49:17.693336    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5449d77-tm4cr" podUID="aacdbcbe-b12e-4969-9cfe-586811191878" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/health\": dial tcp 10.133.0.41:8000: connect: connection refused"
May 25 11:49:27.627805 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:49:27.627768    2576 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b5bb44845g5xgx"
May 25 11:49:27.635378 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:49:27.635350    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b5bb44845g5xgx"
May 25 11:49:27.693232 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:49:27.693186    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5449d77-tm4cr" podUID="aacdbcbe-b12e-4969-9cfe-586811191878" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/health\": dial tcp 10.133.0.41:8000: connect: connection refused"
May 25 11:49:35.074314 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:49:35.074261    2576 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b5bb44845g5xgx"]
May 25 11:49:35.074830 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:49:35.074643    2576 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b5bb44845g5xgx" podUID="ce4cff5d-2132-4b5c-9aea-790191e87b00" containerName="main" containerID="cri-o://71c7ed88b79de20a8167e3bc3e0f2c7a790f58d2bdc3b608685f12a5a710efc6" gracePeriod=30
May 25 11:49:37.693036 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:49:37.692992    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5449d77-tm4cr" podUID="aacdbcbe-b12e-4969-9cfe-586811191878" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/health\": dial tcp 10.133.0.41:8000: connect: connection refused"
May 25 11:49:45.548388 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:49:45.548351    2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"]
May 25 11:49:45.548825 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:49:45.548710    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="887d4410-aee9-44bc-99ee-9fa600b7b6ac" containerName="main"
May 25 11:49:45.548825 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:49:45.548745    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="887d4410-aee9-44bc-99ee-9fa600b7b6ac" containerName="main"
May 25 11:49:45.548825 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:49:45.548768    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="887d4410-aee9-44bc-99ee-9fa600b7b6ac" containerName="storage-initializer"
May 25 11:49:45.548825 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:49:45.548777    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="887d4410-aee9-44bc-99ee-9fa600b7b6ac" containerName="storage-initializer"
May 25 11:49:45.548825 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:49:45.548790    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad589270-6564-4cdd-a936-e645856ca27c" containerName="main"
May 25 11:49:45.548825 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:49:45.548799    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad589270-6564-4cdd-a936-e645856ca27c" containerName="main"
May 25 11:49:45.548825 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:49:45.548808    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e96af524-82a6-41ef-8bf2-8a1bf59bc33f" containerName="main"
May 25 11:49:45.548825 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:49:45.548813    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="e96af524-82a6-41ef-8bf2-8a1bf59bc33f" containerName="main"
May 25 11:49:45.548825 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:49:45.548818    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad589270-6564-4cdd-a936-e645856ca27c" containerName="storage-initializer"
May 25 11:49:45.548825 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:49:45.548824    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad589270-6564-4cdd-a936-e645856ca27c" containerName="storage-initializer"
May 25 11:49:45.549130 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:49:45.548840    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e96af524-82a6-41ef-8bf2-8a1bf59bc33f" containerName="storage-initializer"
May 25 11:49:45.549130 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:49:45.548845    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="e96af524-82a6-41ef-8bf2-8a1bf59bc33f" containerName="storage-initializer"
May 25 11:49:45.549130 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:49:45.548856    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e96af524-82a6-41ef-8bf2-8a1bf59bc33f" containerName="llm-d-routing-sidecar"
May 25 11:49:45.549130 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:49:45.548861    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="e96af524-82a6-41ef-8bf2-8a1bf59bc33f" containerName="llm-d-routing-sidecar"
May 25 11:49:45.549130 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:49:45.548919    2576 memory_manager.go:356] "RemoveStaleState removing state" podUID="e96af524-82a6-41ef-8bf2-8a1bf59bc33f" containerName="main"
May 25 11:49:45.549130 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:49:45.548929    2576 memory_manager.go:356] "RemoveStaleState removing state" podUID="887d4410-aee9-44bc-99ee-9fa600b7b6ac" containerName="main"
May 25 11:49:45.549130 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:49:45.548936    2576 memory_manager.go:356] "RemoveStaleState removing state" podUID="ad589270-6564-4cdd-a936-e645856ca27c" containerName="main"
May 25 11:49:45.549130 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:49:45.548941    2576 memory_manager.go:356] "RemoveStaleState removing state" podUID="e96af524-82a6-41ef-8bf2-8a1bf59bc33f" containerName="llm-d-routing-sidecar"
May 25 11:49:45.553140 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:49:45.553122    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
May 25 11:49:45.555634 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:49:45.555613    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv4e643bc258191ffc517a31cd1d0ddd27-kserve-self-signed-certs\""
May 25 11:49:45.555879 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:49:45.555860    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-fb-opt-125m-route-f312f5-cb7fb8cf-dockercfg-bnwh4\""
May 25 11:49:45.561048 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:49:45.561026    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"]
May 25 11:49:45.651404 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:49:45.651368    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40b68ed6-9a1c-4ac4-929f-cbb932357de5-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"40b68ed6-9a1c-4ac4-929f-cbb932357de5\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
May 25 11:49:45.651577 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:49:45.651411    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/40b68ed6-9a1c-4ac4-929f-cbb932357de5-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"40b68ed6-9a1c-4ac4-929f-cbb932357de5\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
May 25 11:49:45.651577 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:49:45.651441    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/40b68ed6-9a1c-4ac4-929f-cbb932357de5-home\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"40b68ed6-9a1c-4ac4-929f-cbb932357de5\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
May 25 11:49:45.651577 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:49:45.651497    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvz97\" (UniqueName: \"kubernetes.io/projected/40b68ed6-9a1c-4ac4-929f-cbb932357de5-kube-api-access-rvz97\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"40b68ed6-9a1c-4ac4-929f-cbb932357de5\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
May 25 11:49:45.651577 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:49:45.651551    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40b68ed6-9a1c-4ac4-929f-cbb932357de5-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"40b68ed6-9a1c-4ac4-929f-cbb932357de5\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
May 25 11:49:45.651752 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:49:45.651582    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/40b68ed6-9a1c-4ac4-929f-cbb932357de5-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"40b68ed6-9a1c-4ac4-929f-cbb932357de5\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
May 25 11:49:45.651752 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:49:45.651618    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/40b68ed6-9a1c-4ac4-929f-cbb932357de5-dshm\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"40b68ed6-9a1c-4ac4-929f-cbb932357de5\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
May 25 11:49:45.752960 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:49:45.752928    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/40b68ed6-9a1c-4ac4-929f-cbb932357de5-home\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"40b68ed6-9a1c-4ac4-929f-cbb932357de5\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
May 25 11:49:45.753162 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:49:45.752980    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-rvz97\" (UniqueName: \"kubernetes.io/projected/40b68ed6-9a1c-4ac4-929f-cbb932357de5-kube-api-access-rvz97\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"40b68ed6-9a1c-4ac4-929f-cbb932357de5\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
May 25 11:49:45.753162 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:49:45.753014    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40b68ed6-9a1c-4ac4-929f-cbb932357de5-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"40b68ed6-9a1c-4ac4-929f-cbb932357de5\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
May 25 11:49:45.753162 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:49:45.753039    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/40b68ed6-9a1c-4ac4-929f-cbb932357de5-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"40b68ed6-9a1c-4ac4-929f-cbb932357de5\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
May 25 11:49:45.753162 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:49:45.753086    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/40b68ed6-9a1c-4ac4-929f-cbb932357de5-dshm\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"40b68ed6-9a1c-4ac4-929f-cbb932357de5\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
May 25 11:49:45.753425 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:49:45.753175    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40b68ed6-9a1c-4ac4-929f-cbb932357de5-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"40b68ed6-9a1c-4ac4-929f-cbb932357de5\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
May 25 11:49:45.753425 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:49:45.753242    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/40b68ed6-9a1c-4ac4-929f-cbb932357de5-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"40b68ed6-9a1c-4ac4-929f-cbb932357de5\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
May 25 11:49:45.753425 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:49:45.753363    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/40b68ed6-9a1c-4ac4-929f-cbb932357de5-home\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"40b68ed6-9a1c-4ac4-929f-cbb932357de5\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
May 25 11:49:45.753425 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:49:45.753394    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/40b68ed6-9a1c-4ac4-929f-cbb932357de5-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"40b68ed6-9a1c-4ac4-929f-cbb932357de5\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
May 25 11:49:45.753640 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:49:45.753465    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40b68ed6-9a1c-4ac4-929f-cbb932357de5-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"40b68ed6-9a1c-4ac4-929f-cbb932357de5\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
May 25 11:49:45.753965 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:49:45.753939    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/40b68ed6-9a1c-4ac4-929f-cbb932357de5-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"40b68ed6-9a1c-4ac4-929f-cbb932357de5\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
May 25 11:49:45.755485 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:49:45.755460    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/40b68ed6-9a1c-4ac4-929f-cbb932357de5-dshm\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"40b68ed6-9a1c-4ac4-929f-cbb932357de5\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
May 25 11:49:45.755678 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:49:45.755657    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40b68ed6-9a1c-4ac4-929f-cbb932357de5-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"40b68ed6-9a1c-4ac4-929f-cbb932357de5\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
May 25 11:49:45.760671 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:49:45.760645    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvz97\" (UniqueName: \"kubernetes.io/projected/40b68ed6-9a1c-4ac4-929f-cbb932357de5-kube-api-access-rvz97\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"40b68ed6-9a1c-4ac4-929f-cbb932357de5\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
May 25 11:49:45.864497 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:49:45.864414    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
May 25 11:49:46.009059 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:49:46.009029    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"]
May 25 11:49:46.011169 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:49:46.011139    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40b68ed6_9a1c_4ac4_929f_cbb932357de5.slice/crio-d2580a9510ca9de74d92a734e335da3bae5a3ba13cf7f2bef81975894110ac4b WatchSource:0}: Error finding container d2580a9510ca9de74d92a734e335da3bae5a3ba13cf7f2bef81975894110ac4b: Status 404 returned error can't find the container with id d2580a9510ca9de74d92a734e335da3bae5a3ba13cf7f2bef81975894110ac4b
May 25 11:49:46.848143 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:49:46.848102    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" event={"ID":"40b68ed6-9a1c-4ac4-929f-cbb932357de5","Type":"ContainerStarted","Data":"9384d5f7cd456e9180b76e0ed2fa2ca3884686694172cbae00c25ce9b14bbc19"}
May 25 11:49:46.848646 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:49:46.848151    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" event={"ID":"40b68ed6-9a1c-4ac4-929f-cbb932357de5","Type":"ContainerStarted","Data":"d2580a9510ca9de74d92a734e335da3bae5a3ba13cf7f2bef81975894110ac4b"}
May 25 11:49:47.693603 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:49:47.693549    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5449d77-tm4cr" podUID="aacdbcbe-b12e-4969-9cfe-586811191878" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/health\": dial tcp 10.133.0.41:8000: connect: connection refused"
May 25 11:49:57.693105 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:49:57.693048    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5449d77-tm4cr" podUID="aacdbcbe-b12e-4969-9cfe-586811191878" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/health\": dial tcp 10.133.0.41:8000: connect: connection refused"
May 25 11:50:02.905540 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:50:02.905458    2576 generic.go:358] "Generic (PLEG): container finished" podID="40b68ed6-9a1c-4ac4-929f-cbb932357de5" containerID="9384d5f7cd456e9180b76e0ed2fa2ca3884686694172cbae00c25ce9b14bbc19" exitCode=0
May 25 11:50:02.905908 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:50:02.905531    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" event={"ID":"40b68ed6-9a1c-4ac4-929f-cbb932357de5","Type":"ContainerDied","Data":"9384d5f7cd456e9180b76e0ed2fa2ca3884686694172cbae00c25ce9b14bbc19"}
May 25 11:50:03.911598 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:50:03.911553    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" event={"ID":"40b68ed6-9a1c-4ac4-929f-cbb932357de5","Type":"ContainerStarted","Data":"6627ee637185c146c42e28288e7bbf36e5252340fe63ae43212010331fac60f7"}
May 25 11:50:03.931357 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:50:03.931301    2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" podStartSLOduration=18.931263863 podStartE2EDuration="18.931263863s" podCreationTimestamp="2026-05-25 11:49:45 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 11:50:03.930114071 +0000 UTC m=+1949.447845511" watchObservedRunningTime="2026-05-25 11:50:03.931263863 +0000 UTC m=+1949.448995327"
May 25 11:50:05.346016 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:50:05.345967    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b5bb44845g5xgx_ce4cff5d-2132-4b5c-9aea-790191e87b00/main/0.log"
May 25 11:50:05.346500 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:50:05.346483    2576 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b5bb44845g5xgx"
May 25 11:50:05.445582 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:50:05.445548    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ce4cff5d-2132-4b5c-9aea-790191e87b00-kserve-provision-location\") pod \"ce4cff5d-2132-4b5c-9aea-790191e87b00\" (UID: \"ce4cff5d-2132-4b5c-9aea-790191e87b00\") "
May 25 11:50:05.445777 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:50:05.445653    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce4cff5d-2132-4b5c-9aea-790191e87b00-tls-certs\") pod \"ce4cff5d-2132-4b5c-9aea-790191e87b00\" (UID: \"ce4cff5d-2132-4b5c-9aea-790191e87b00\") "
May 25 11:50:05.445777 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:50:05.445694    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce4cff5d-2132-4b5c-9aea-790191e87b00-model-cache\") pod \"ce4cff5d-2132-4b5c-9aea-790191e87b00\" (UID: \"ce4cff5d-2132-4b5c-9aea-790191e87b00\") "
May 25 11:50:05.445777 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:50:05.445727    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ce4cff5d-2132-4b5c-9aea-790191e87b00-home\") pod \"ce4cff5d-2132-4b5c-9aea-790191e87b00\" (UID: \"ce4cff5d-2132-4b5c-9aea-790191e87b00\") "
May 25 11:50:05.445777 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:50:05.445762    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce4cff5d-2132-4b5c-9aea-790191e87b00-tmp-dir\") pod \"ce4cff5d-2132-4b5c-9aea-790191e87b00\" (UID: \"ce4cff5d-2132-4b5c-9aea-790191e87b00\") "
May 25 11:50:05.446056 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:50:05.445795    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce4cff5d-2132-4b5c-9aea-790191e87b00-dshm\") pod \"ce4cff5d-2132-4b5c-9aea-790191e87b00\" (UID: \"ce4cff5d-2132-4b5c-9aea-790191e87b00\") "
May 25 11:50:05.446056 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:50:05.445821    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qjvzz\" (UniqueName: \"kubernetes.io/projected/ce4cff5d-2132-4b5c-9aea-790191e87b00-kube-api-access-qjvzz\") pod \"ce4cff5d-2132-4b5c-9aea-790191e87b00\" (UID: \"ce4cff5d-2132-4b5c-9aea-790191e87b00\") "
May 25 11:50:05.446190 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:50:05.446163    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce4cff5d-2132-4b5c-9aea-790191e87b00-model-cache" (OuterVolumeSpecName: "model-cache") pod "ce4cff5d-2132-4b5c-9aea-790191e87b00" (UID: "ce4cff5d-2132-4b5c-9aea-790191e87b00"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:50:05.447477 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:50:05.447432    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce4cff5d-2132-4b5c-9aea-790191e87b00-home" (OuterVolumeSpecName: "home") pod "ce4cff5d-2132-4b5c-9aea-790191e87b00" (UID: "ce4cff5d-2132-4b5c-9aea-790191e87b00"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:50:05.449859 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:50:05.449828    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce4cff5d-2132-4b5c-9aea-790191e87b00-dshm" (OuterVolumeSpecName: "dshm") pod "ce4cff5d-2132-4b5c-9aea-790191e87b00" (UID: "ce4cff5d-2132-4b5c-9aea-790191e87b00"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:50:05.449971 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:50:05.449896    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce4cff5d-2132-4b5c-9aea-790191e87b00-kube-api-access-qjvzz" (OuterVolumeSpecName: "kube-api-access-qjvzz") pod "ce4cff5d-2132-4b5c-9aea-790191e87b00" (UID: "ce4cff5d-2132-4b5c-9aea-790191e87b00"). InnerVolumeSpecName "kube-api-access-qjvzz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 11:50:05.450792 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:50:05.450769    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce4cff5d-2132-4b5c-9aea-790191e87b00-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ce4cff5d-2132-4b5c-9aea-790191e87b00" (UID: "ce4cff5d-2132-4b5c-9aea-790191e87b00"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 11:50:05.475340 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:50:05.475261    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce4cff5d-2132-4b5c-9aea-790191e87b00-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ce4cff5d-2132-4b5c-9aea-790191e87b00" (UID: "ce4cff5d-2132-4b5c-9aea-790191e87b00"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:50:05.529425 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:50:05.529386    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce4cff5d-2132-4b5c-9aea-790191e87b00-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ce4cff5d-2132-4b5c-9aea-790191e87b00" (UID: "ce4cff5d-2132-4b5c-9aea-790191e87b00"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:50:05.547338 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:50:05.547309    2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ce4cff5d-2132-4b5c-9aea-790191e87b00-kserve-provision-location\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
May 25 11:50:05.547338 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:50:05.547339    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce4cff5d-2132-4b5c-9aea-790191e87b00-tls-certs\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
May 25 11:50:05.547538 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:50:05.547355    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce4cff5d-2132-4b5c-9aea-790191e87b00-model-cache\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
May 25 11:50:05.547538 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:50:05.547367    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ce4cff5d-2132-4b5c-9aea-790191e87b00-home\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
May 25 11:50:05.547538 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:50:05.547379    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce4cff5d-2132-4b5c-9aea-790191e87b00-tmp-dir\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
May 25 11:50:05.547538 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:50:05.547390    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce4cff5d-2132-4b5c-9aea-790191e87b00-dshm\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
May 25 11:50:05.547538 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:50:05.547401    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qjvzz\" (UniqueName: \"kubernetes.io/projected/ce4cff5d-2132-4b5c-9aea-790191e87b00-kube-api-access-qjvzz\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
May 25 11:50:05.865208 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:50:05.865167    2576 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
May 25 11:50:05.866943 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:50:05.866909    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" podUID="40b68ed6-9a1c-4ac4-929f-cbb932357de5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/health\": dial tcp 10.133.0.43:8000: connect: connection refused"
May 25 11:50:05.919242 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:50:05.919213    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b5bb44845g5xgx_ce4cff5d-2132-4b5c-9aea-790191e87b00/main/0.log"
May 25 11:50:05.919606 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:50:05.919578    2576 generic.go:358] "Generic (PLEG): container finished" podID="ce4cff5d-2132-4b5c-9aea-790191e87b00" containerID="71c7ed88b79de20a8167e3bc3e0f2c7a790f58d2bdc3b608685f12a5a710efc6" exitCode=137
May 25 11:50:05.919747 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:50:05.919670    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b5bb44845g5xgx" event={"ID":"ce4cff5d-2132-4b5c-9aea-790191e87b00","Type":"ContainerDied","Data":"71c7ed88b79de20a8167e3bc3e0f2c7a790f58d2bdc3b608685f12a5a710efc6"}
May 25 11:50:05.919747 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:50:05.919703    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b5bb44845g5xgx" event={"ID":"ce4cff5d-2132-4b5c-9aea-790191e87b00","Type":"ContainerDied","Data":"8bd2b9f0215d5d9d063c6f25d3c9f6c95e6bdc9213a5ec846211ad9ac422a248"}
May 25 11:50:05.919747 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:50:05.919722    2576 scope.go:117] "RemoveContainer" containerID="71c7ed88b79de20a8167e3bc3e0f2c7a790f58d2bdc3b608685f12a5a710efc6"
May 25 11:50:05.919921 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:50:05.919676    2576 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b5bb44845g5xgx"
May 25 11:50:05.929987 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:50:05.929970    2576 scope.go:117] "RemoveContainer" containerID="9d64f4e18ba53c5fd64d0a1a9f0ccce86b55fdd178d4169b2320620b3db4d8da"
May 25 11:50:05.944586 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:50:05.944542    2576 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b5bb44845g5xgx"]
May 25 11:50:05.947400 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:50:05.947379    2576 scope.go:117] "RemoveContainer" containerID="71c7ed88b79de20a8167e3bc3e0f2c7a790f58d2bdc3b608685f12a5a710efc6"
May 25 11:50:05.947699 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:50:05.947680    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71c7ed88b79de20a8167e3bc3e0f2c7a790f58d2bdc3b608685f12a5a710efc6\": container with ID starting with 71c7ed88b79de20a8167e3bc3e0f2c7a790f58d2bdc3b608685f12a5a710efc6 not found: ID does not exist" containerID="71c7ed88b79de20a8167e3bc3e0f2c7a790f58d2bdc3b608685f12a5a710efc6"
May 25 11:50:05.947773 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:50:05.947708    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71c7ed88b79de20a8167e3bc3e0f2c7a790f58d2bdc3b608685f12a5a710efc6"} err="failed to get container status \"71c7ed88b79de20a8167e3bc3e0f2c7a790f58d2bdc3b608685f12a5a710efc6\": rpc error: code = NotFound desc = could not find container \"71c7ed88b79de20a8167e3bc3e0f2c7a790f58d2bdc3b608685f12a5a710efc6\": container with ID starting with 71c7ed88b79de20a8167e3bc3e0f2c7a790f58d2bdc3b608685f12a5a710efc6 not found: ID does not exist"
May 25 11:50:05.947773 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:50:05.947730    2576 scope.go:117] "RemoveContainer" containerID="9d64f4e18ba53c5fd64d0a1a9f0ccce86b55fdd178d4169b2320620b3db4d8da"
May 25 11:50:05.948004 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:50:05.947983    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d64f4e18ba53c5fd64d0a1a9f0ccce86b55fdd178d4169b2320620b3db4d8da\": container with ID starting with 9d64f4e18ba53c5fd64d0a1a9f0ccce86b55fdd178d4169b2320620b3db4d8da not found: ID does not exist" containerID="9d64f4e18ba53c5fd64d0a1a9f0ccce86b55fdd178d4169b2320620b3db4d8da"
May 25 11:50:05.948114 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:50:05.948015    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d64f4e18ba53c5fd64d0a1a9f0ccce86b55fdd178d4169b2320620b3db4d8da"} err="failed to get container status \"9d64f4e18ba53c5fd64d0a1a9f0ccce86b55fdd178d4169b2320620b3db4d8da\": rpc error: code = NotFound desc = could not find container \"9d64f4e18ba53c5fd64d0a1a9f0ccce86b55fdd178d4169b2320620b3db4d8da\": container with ID starting with 9d64f4e18ba53c5fd64d0a1a9f0ccce86b55fdd178d4169b2320620b3db4d8da not found: ID does not exist"
May 25 11:50:05.949117 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:50:05.949099    2576 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b5bb44845g5xgx"]
May 25 11:50:07.098545 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:50:07.098509    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ce4cff5d-2132-4b5c-9aea-790191e87b00" path="/var/lib/kubelet/pods/ce4cff5d-2132-4b5c-9aea-790191e87b00/volumes"
May 25 11:50:07.703047 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:50:07.703017    2576 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5449d77-tm4cr"
May 25 11:50:07.710711 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:50:07.710688    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5449d77-tm4cr"
May 25 11:50:08.769202 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:50:08.769169    2576 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-5449d77-tm4cr"]
May 25 11:50:08.931171 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:50:08.931121    2576 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5449d77-tm4cr" podUID="aacdbcbe-b12e-4969-9cfe-586811191878" containerName="main" containerID="cri-o://cc2b8e06e66a0a952f0a4d67be77249cfdbec0fac200d7b8c83dd5f5af544f48" gracePeriod=30
May 25 11:50:15.864937 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:50:15.864906    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
May 25 11:50:15.865401 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:50:15.865200    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" podUID="40b68ed6-9a1c-4ac4-929f-cbb932357de5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/health\": dial tcp 10.133.0.43:8000: connect: connection refused"
May 25 11:50:18.593056 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:50:18.592977    2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-5449d77-6g8hv"]
May 25 11:50:18.593493 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:50:18.593473    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce4cff5d-2132-4b5c-9aea-790191e87b00" containerName="main"
May 25 11:50:18.593539 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:50:18.593496    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce4cff5d-2132-4b5c-9aea-790191e87b00" containerName="main"
May 25 11:50:18.593574 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:50:18.593546    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce4cff5d-2132-4b5c-9aea-790191e87b00" containerName="storage-initializer"
May 25 11:50:18.593574 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:50:18.593555    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce4cff5d-2132-4b5c-9aea-790191e87b00" containerName="storage-initializer"
May 25 11:50:18.593637 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:50:18.593626    2576 memory_manager.go:356] "RemoveStaleState removing state" podUID="ce4cff5d-2132-4b5c-9aea-790191e87b00" containerName="main"
May 25 11:50:18.597008 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:50:18.596989    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5449d77-6g8hv"
May 25 11:50:18.615802 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:50:18.615770    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-5449d77-6g8hv"]
May 25 11:50:18.664850 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:50:18.664815    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1c2857a2-c79b-41c3-a27d-01258dd8fdca-kserve-provision-location\") pod \"stop-feature-test-kserve-5449d77-6g8hv\" (UID: \"1c2857a2-c79b-41c3-a27d-01258dd8fdca\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5449d77-6g8hv"
May 25 11:50:18.664977 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:50:18.664869    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c2857a2-c79b-41c3-a27d-01258dd8fdca-dshm\") pod \"stop-feature-test-kserve-5449d77-6g8hv\" (UID: \"1c2857a2-c79b-41c3-a27d-01258dd8fdca\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5449d77-6g8hv"
May 25 11:50:18.664977 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:50:18.664938    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c2857a2-c79b-41c3-a27d-01258dd8fdca-model-cache\") pod \"stop-feature-test-kserve-5449d77-6g8hv\" (UID: \"1c2857a2-c79b-41c3-a27d-01258dd8fdca\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5449d77-6g8hv"
May 25 11:50:18.665079 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:50:18.664979    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c2857a2-c79b-41c3-a27d-01258dd8fdca-tmp-dir\") pod \"stop-feature-test-kserve-5449d77-6g8hv\" (UID: \"1c2857a2-c79b-41c3-a27d-01258dd8fdca\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5449d77-6g8hv"
May 25 11:50:18.665079 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:50:18.665014    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c2857a2-c79b-41c3-a27d-01258dd8fdca-home\") pod \"stop-feature-test-kserve-5449d77-6g8hv\" (UID: \"1c2857a2-c79b-41c3-a27d-01258dd8fdca\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5449d77-6g8hv"
May 25 11:50:18.665079 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:50:18.665036    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c2857a2-c79b-41c3-a27d-01258dd8fdca-tls-certs\") pod \"stop-feature-test-kserve-5449d77-6g8hv\" (UID: \"1c2857a2-c79b-41c3-a27d-01258dd8fdca\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5449d77-6g8hv"
May 25 11:50:18.665192 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:50:18.665121    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkgj4\" (UniqueName: \"kubernetes.io/projected/1c2857a2-c79b-41c3-a27d-01258dd8fdca-kube-api-access-pkgj4\") pod \"stop-feature-test-kserve-5449d77-6g8hv\" (UID: \"1c2857a2-c79b-41c3-a27d-01258dd8fdca\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5449d77-6g8hv"
May 25 11:50:18.766320 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:50:18.766257    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1c2857a2-c79b-41c3-a27d-01258dd8fdca-kserve-provision-location\") pod \"stop-feature-test-kserve-5449d77-6g8hv\" (UID: \"1c2857a2-c79b-41c3-a27d-01258dd8fdca\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5449d77-6g8hv"
May 25 11:50:18.766480 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:50:18.766358    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c2857a2-c79b-41c3-a27d-01258dd8fdca-dshm\") pod \"stop-feature-test-kserve-5449d77-6g8hv\" (UID: \"1c2857a2-c79b-41c3-a27d-01258dd8fdca\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5449d77-6g8hv"
May 25 11:50:18.766480 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:50:18.766410    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c2857a2-c79b-41c3-a27d-01258dd8fdca-model-cache\") pod \"stop-feature-test-kserve-5449d77-6g8hv\" (UID: \"1c2857a2-c79b-41c3-a27d-01258dd8fdca\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5449d77-6g8hv"
May 25 11:50:18.766480 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:50:18.766444    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c2857a2-c79b-41c3-a27d-01258dd8fdca-tmp-dir\") pod \"stop-feature-test-kserve-5449d77-6g8hv\" (UID: \"1c2857a2-c79b-41c3-a27d-01258dd8fdca\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5449d77-6g8hv"
May 25 11:50:18.766644 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:50:18.766480    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c2857a2-c79b-41c3-a27d-01258dd8fdca-home\") pod \"stop-feature-test-kserve-5449d77-6g8hv\" (UID: \"1c2857a2-c79b-41c3-a27d-01258dd8fdca\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5449d77-6g8hv"
May 25 11:50:18.766644 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:50:18.766506    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c2857a2-c79b-41c3-a27d-01258dd8fdca-tls-certs\") pod \"stop-feature-test-kserve-5449d77-6g8hv\" (UID: \"1c2857a2-c79b-41c3-a27d-01258dd8fdca\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5449d77-6g8hv"
May 25 11:50:18.766644 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:50:18.766602    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1c2857a2-c79b-41c3-a27d-01258dd8fdca-kserve-provision-location\") pod \"stop-feature-test-kserve-5449d77-6g8hv\" (UID: \"1c2857a2-c79b-41c3-a27d-01258dd8fdca\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5449d77-6g8hv"
May 25 11:50:18.766644 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:50:18.766631    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-pkgj4\" (UniqueName: \"kubernetes.io/projected/1c2857a2-c79b-41c3-a27d-01258dd8fdca-kube-api-access-pkgj4\") pod \"stop-feature-test-kserve-5449d77-6g8hv\" (UID: \"1c2857a2-c79b-41c3-a27d-01258dd8fdca\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5449d77-6g8hv"
May 25 11:50:18.766958 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:50:18.766933    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c2857a2-c79b-41c3-a27d-01258dd8fdca-home\") pod \"stop-feature-test-kserve-5449d77-6g8hv\" (UID: \"1c2857a2-c79b-41c3-a27d-01258dd8fdca\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5449d77-6g8hv"
May 25 11:50:18.767102 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:50:18.766973    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c2857a2-c79b-41c3-a27d-01258dd8fdca-tmp-dir\") pod \"stop-feature-test-kserve-5449d77-6g8hv\" (UID: \"1c2857a2-c79b-41c3-a27d-01258dd8fdca\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5449d77-6g8hv"
May 25 11:50:18.767102 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:50:18.766992    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c2857a2-c79b-41c3-a27d-01258dd8fdca-model-cache\") pod \"stop-feature-test-kserve-5449d77-6g8hv\" (UID: \"1c2857a2-c79b-41c3-a27d-01258dd8fdca\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5449d77-6g8hv"
May 25 11:50:18.768593 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:50:18.768570    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c2857a2-c79b-41c3-a27d-01258dd8fdca-dshm\") pod \"stop-feature-test-kserve-5449d77-6g8hv\" (UID: \"1c2857a2-c79b-41c3-a27d-01258dd8fdca\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5449d77-6g8hv"
May 25 11:50:18.768907 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:50:18.768889    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c2857a2-c79b-41c3-a27d-01258dd8fdca-tls-certs\") pod \"stop-feature-test-kserve-5449d77-6g8hv\" (UID: \"1c2857a2-c79b-41c3-a27d-01258dd8fdca\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5449d77-6g8hv"
May 25 11:50:18.775214 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:50:18.775195    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkgj4\" (UniqueName: \"kubernetes.io/projected/1c2857a2-c79b-41c3-a27d-01258dd8fdca-kube-api-access-pkgj4\") pod \"stop-feature-test-kserve-5449d77-6g8hv\" (UID: \"1c2857a2-c79b-41c3-a27d-01258dd8fdca\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5449d77-6g8hv"
May 25 11:50:18.907822 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:50:18.907733    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5449d77-6g8hv"
May 25 11:50:19.052491 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:50:19.052404    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-5449d77-6g8hv"]
May 25 11:50:19.055040 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:50:19.055011    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c2857a2_c79b_41c3_a27d_01258dd8fdca.slice/crio-4051442f2423ef5c3446a9df098d1909be289467624b68b25d5d999cba15bf53 WatchSource:0}: Error finding container 4051442f2423ef5c3446a9df098d1909be289467624b68b25d5d999cba15bf53: Status 404 returned error can't find the container with id 4051442f2423ef5c3446a9df098d1909be289467624b68b25d5d999cba15bf53
May 25 11:50:19.970846 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:50:19.970799    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5449d77-6g8hv" event={"ID":"1c2857a2-c79b-41c3-a27d-01258dd8fdca","Type":"ContainerStarted","Data":"e6ef237aad33fca793549ca9b90cb70d131ba5fff58de97ee935dbd56a8930b8"}
May 25 11:50:19.971339 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:50:19.970853    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5449d77-6g8hv" event={"ID":"1c2857a2-c79b-41c3-a27d-01258dd8fdca","Type":"ContainerStarted","Data":"4051442f2423ef5c3446a9df098d1909be289467624b68b25d5d999cba15bf53"}
May 25 11:50:25.865391 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:50:25.865341    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" podUID="40b68ed6-9a1c-4ac4-929f-cbb932357de5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/health\": dial tcp 10.133.0.43:8000: connect: connection refused"
May 25 11:50:35.865477 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:50:35.865434    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" podUID="40b68ed6-9a1c-4ac4-929f-cbb932357de5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/health\": dial tcp 10.133.0.43:8000: connect: connection refused"
May 25 11:50:39.187388 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:50:39.187330    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_stop-feature-test-kserve-5449d77-tm4cr_aacdbcbe-b12e-4969-9cfe-586811191878/main/0.log"
May 25 11:50:39.187710 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:50:39.187675    2576 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5449d77-tm4cr"
May 25 11:50:39.352822 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:50:39.352790    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aacdbcbe-b12e-4969-9cfe-586811191878-model-cache\") pod \"aacdbcbe-b12e-4969-9cfe-586811191878\" (UID: \"aacdbcbe-b12e-4969-9cfe-586811191878\") "
May 25 11:50:39.352990 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:50:39.352885    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-crtvg\" (UniqueName: \"kubernetes.io/projected/aacdbcbe-b12e-4969-9cfe-586811191878-kube-api-access-crtvg\") pod \"aacdbcbe-b12e-4969-9cfe-586811191878\" (UID: \"aacdbcbe-b12e-4969-9cfe-586811191878\") "
May 25 11:50:39.352990 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:50:39.352907    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aacdbcbe-b12e-4969-9cfe-586811191878-home\") pod \"aacdbcbe-b12e-4969-9cfe-586811191878\" (UID: \"aacdbcbe-b12e-4969-9cfe-586811191878\") "
May 25 11:50:39.352990 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:50:39.352936    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aacdbcbe-b12e-4969-9cfe-586811191878-dshm\") pod \"aacdbcbe-b12e-4969-9cfe-586811191878\" (UID: \"aacdbcbe-b12e-4969-9cfe-586811191878\") "
May 25 11:50:39.352990 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:50:39.352959    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aacdbcbe-b12e-4969-9cfe-586811191878-tmp-dir\") pod \"aacdbcbe-b12e-4969-9cfe-586811191878\" (UID: \"aacdbcbe-b12e-4969-9cfe-586811191878\") "
May 25 11:50:39.353213 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:50:39.353002    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aacdbcbe-b12e-4969-9cfe-586811191878-tls-certs\") pod \"aacdbcbe-b12e-4969-9cfe-586811191878\" (UID: \"aacdbcbe-b12e-4969-9cfe-586811191878\") "
May 25 11:50:39.353213 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:50:39.353025    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/aacdbcbe-b12e-4969-9cfe-586811191878-kserve-provision-location\") pod \"aacdbcbe-b12e-4969-9cfe-586811191878\" (UID: \"aacdbcbe-b12e-4969-9cfe-586811191878\") "
May 25 11:50:39.353213 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:50:39.353116    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aacdbcbe-b12e-4969-9cfe-586811191878-model-cache" (OuterVolumeSpecName: "model-cache") pod "aacdbcbe-b12e-4969-9cfe-586811191878" (UID: "aacdbcbe-b12e-4969-9cfe-586811191878"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:50:39.353421 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:50:39.353326    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aacdbcbe-b12e-4969-9cfe-586811191878-model-cache\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
May 25 11:50:39.353594 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:50:39.353565    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aacdbcbe-b12e-4969-9cfe-586811191878-home" (OuterVolumeSpecName: "home") pod "aacdbcbe-b12e-4969-9cfe-586811191878" (UID: "aacdbcbe-b12e-4969-9cfe-586811191878"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:50:39.355203 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:50:39.355175    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aacdbcbe-b12e-4969-9cfe-586811191878-dshm" (OuterVolumeSpecName: "dshm") pod "aacdbcbe-b12e-4969-9cfe-586811191878" (UID: "aacdbcbe-b12e-4969-9cfe-586811191878"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:50:39.355567 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:50:39.355534    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aacdbcbe-b12e-4969-9cfe-586811191878-kube-api-access-crtvg" (OuterVolumeSpecName: "kube-api-access-crtvg") pod "aacdbcbe-b12e-4969-9cfe-586811191878" (UID: "aacdbcbe-b12e-4969-9cfe-586811191878"). InnerVolumeSpecName "kube-api-access-crtvg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 11:50:39.355684 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:50:39.355572    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aacdbcbe-b12e-4969-9cfe-586811191878-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "aacdbcbe-b12e-4969-9cfe-586811191878" (UID: "aacdbcbe-b12e-4969-9cfe-586811191878"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 11:50:39.371570 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:50:39.371538    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aacdbcbe-b12e-4969-9cfe-586811191878-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "aacdbcbe-b12e-4969-9cfe-586811191878" (UID: "aacdbcbe-b12e-4969-9cfe-586811191878"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:50:39.422916 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:50:39.422874    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aacdbcbe-b12e-4969-9cfe-586811191878-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "aacdbcbe-b12e-4969-9cfe-586811191878" (UID: "aacdbcbe-b12e-4969-9cfe-586811191878"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:50:39.454161 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:50:39.454073    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-crtvg\" (UniqueName: \"kubernetes.io/projected/aacdbcbe-b12e-4969-9cfe-586811191878-kube-api-access-crtvg\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
May 25 11:50:39.454161 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:50:39.454106    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aacdbcbe-b12e-4969-9cfe-586811191878-home\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
May 25 11:50:39.454161 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:50:39.454115    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aacdbcbe-b12e-4969-9cfe-586811191878-dshm\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
May 25 11:50:39.454161 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:50:39.454124    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aacdbcbe-b12e-4969-9cfe-586811191878-tmp-dir\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
May 25 11:50:39.454161 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:50:39.454134    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aacdbcbe-b12e-4969-9cfe-586811191878-tls-certs\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
May 25 11:50:39.454161 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:50:39.454143    2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/aacdbcbe-b12e-4969-9cfe-586811191878-kserve-provision-location\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
May 25 11:50:40.038996 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:50:40.038965    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_stop-feature-test-kserve-5449d77-tm4cr_aacdbcbe-b12e-4969-9cfe-586811191878/main/0.log"
May 25 11:50:40.039305 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:50:40.039256    2576 generic.go:358] "Generic (PLEG): container finished" podID="aacdbcbe-b12e-4969-9cfe-586811191878" containerID="cc2b8e06e66a0a952f0a4d67be77249cfdbec0fac200d7b8c83dd5f5af544f48" exitCode=137
May 25 11:50:40.039418 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:50:40.039339    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5449d77-tm4cr" event={"ID":"aacdbcbe-b12e-4969-9cfe-586811191878","Type":"ContainerDied","Data":"cc2b8e06e66a0a952f0a4d67be77249cfdbec0fac200d7b8c83dd5f5af544f48"}
May 25 11:50:40.039418 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:50:40.039376    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5449d77-tm4cr" event={"ID":"aacdbcbe-b12e-4969-9cfe-586811191878","Type":"ContainerDied","Data":"7d9a90da8986443583dc9365efe51bb28f4c9957d6341143dbd19d4916d737b3"}
May 25 11:50:40.039418 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:50:40.039393    2576 scope.go:117] "RemoveContainer" containerID="cc2b8e06e66a0a952f0a4d67be77249cfdbec0fac200d7b8c83dd5f5af544f48"
May 25 11:50:40.039586 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:50:40.039353    2576 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5449d77-tm4cr"
May 25 11:50:40.049470 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:50:40.049450    2576 scope.go:117] "RemoveContainer" containerID="040b762224fcfa7e740f1a88d058ff4f7a4ed8007612d1ddef1786617e90eb36"
May 25 11:50:40.059972 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:50:40.059954    2576 scope.go:117] "RemoveContainer" containerID="cc2b8e06e66a0a952f0a4d67be77249cfdbec0fac200d7b8c83dd5f5af544f48"
May 25 11:50:40.060202 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:50:40.060184    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc2b8e06e66a0a952f0a4d67be77249cfdbec0fac200d7b8c83dd5f5af544f48\": container with ID starting with cc2b8e06e66a0a952f0a4d67be77249cfdbec0fac200d7b8c83dd5f5af544f48 not found: ID does not exist" containerID="cc2b8e06e66a0a952f0a4d67be77249cfdbec0fac200d7b8c83dd5f5af544f48"
May 25 11:50:40.060261 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:50:40.060209    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc2b8e06e66a0a952f0a4d67be77249cfdbec0fac200d7b8c83dd5f5af544f48"} err="failed to get container status \"cc2b8e06e66a0a952f0a4d67be77249cfdbec0fac200d7b8c83dd5f5af544f48\": rpc error: code = NotFound desc = could not find container \"cc2b8e06e66a0a952f0a4d67be77249cfdbec0fac200d7b8c83dd5f5af544f48\": container with ID starting with cc2b8e06e66a0a952f0a4d67be77249cfdbec0fac200d7b8c83dd5f5af544f48 not found: ID does not exist"
May 25 11:50:40.060261 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:50:40.060226    2576 scope.go:117] "RemoveContainer" containerID="040b762224fcfa7e740f1a88d058ff4f7a4ed8007612d1ddef1786617e90eb36"
May 25 11:50:40.060442 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:50:40.060426    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"040b762224fcfa7e740f1a88d058ff4f7a4ed8007612d1ddef1786617e90eb36\": container with ID starting with 040b762224fcfa7e740f1a88d058ff4f7a4ed8007612d1ddef1786617e90eb36 not found: ID does not exist" containerID="040b762224fcfa7e740f1a88d058ff4f7a4ed8007612d1ddef1786617e90eb36"
May 25 11:50:40.060489 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:50:40.060447    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"040b762224fcfa7e740f1a88d058ff4f7a4ed8007612d1ddef1786617e90eb36"} err="failed to get container status \"040b762224fcfa7e740f1a88d058ff4f7a4ed8007612d1ddef1786617e90eb36\": rpc error: code = NotFound desc = could not find container \"040b762224fcfa7e740f1a88d058ff4f7a4ed8007612d1ddef1786617e90eb36\": container with ID starting with 040b762224fcfa7e740f1a88d058ff4f7a4ed8007612d1ddef1786617e90eb36 not found: ID does not exist"
May 25 11:50:40.064235 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:50:40.064214    2576 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-5449d77-tm4cr"]
May 25 11:50:40.067546 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:50:40.067525    2576 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-5449d77-tm4cr"]
May 25 11:50:41.098218 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:50:41.098186    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aacdbcbe-b12e-4969-9cfe-586811191878" path="/var/lib/kubelet/pods/aacdbcbe-b12e-4969-9cfe-586811191878/volumes"
May 25 11:50:45.865511 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:50:45.865471    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" podUID="40b68ed6-9a1c-4ac4-929f-cbb932357de5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/health\": dial tcp 10.133.0.43:8000: connect: connection refused"
May 25 11:50:55.865000 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:50:55.864960    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" podUID="40b68ed6-9a1c-4ac4-929f-cbb932357de5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/health\": dial tcp 10.133.0.43:8000: connect: connection refused"
May 25 11:51:05.865761 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:51:05.865708    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" podUID="40b68ed6-9a1c-4ac4-929f-cbb932357de5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/health\": dial tcp 10.133.0.43:8000: connect: connection refused"
May 25 11:51:15.865827 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:51:15.865776    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" podUID="40b68ed6-9a1c-4ac4-929f-cbb932357de5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/health\": dial tcp 10.133.0.43:8000: connect: connection refused"
May 25 11:51:25.865401 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:51:25.865350    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" podUID="40b68ed6-9a1c-4ac4-929f-cbb932357de5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/health\": dial tcp 10.133.0.43:8000: connect: connection refused"
May 25 11:51:35.232081 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:51:35.232046    2576 generic.go:358] "Generic (PLEG): container finished" podID="1c2857a2-c79b-41c3-a27d-01258dd8fdca" containerID="e6ef237aad33fca793549ca9b90cb70d131ba5fff58de97ee935dbd56a8930b8" exitCode=0
May 25 11:51:35.232599 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:51:35.232123    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5449d77-6g8hv" event={"ID":"1c2857a2-c79b-41c3-a27d-01258dd8fdca","Type":"ContainerDied","Data":"e6ef237aad33fca793549ca9b90cb70d131ba5fff58de97ee935dbd56a8930b8"}
May 25 11:51:35.875097 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:51:35.875069    2576 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
May 25 11:51:35.887228 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:51:35.887190    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
May 25 11:51:36.237609 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:51:36.237572    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5449d77-6g8hv" event={"ID":"1c2857a2-c79b-41c3-a27d-01258dd8fdca","Type":"ContainerStarted","Data":"a6f6da7b97d853cf0bab1ae707ef49547d4d5bfe14e2a36652aea82eeb38a546"}
May 25 11:51:36.261071 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:51:36.261012    2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5449d77-6g8hv" podStartSLOduration=78.260993923 podStartE2EDuration="1m18.260993923s" podCreationTimestamp="2026-05-25 11:50:18 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 11:51:36.260133914 +0000 UTC m=+2041.777865370" watchObservedRunningTime="2026-05-25 11:51:36.260993923 +0000 UTC m=+2041.778725380"
May 25 11:51:38.907936 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:51:38.907886    2576 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5449d77-6g8hv"
May 25 11:51:38.908381 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:51:38.907979    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5449d77-6g8hv"
May 25 11:51:38.909592 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:51:38.909564    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5449d77-6g8hv" podUID="1c2857a2-c79b-41c3-a27d-01258dd8fdca" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/health\": dial tcp 10.133.0.44:8000: connect: connection refused"
May 25 11:51:43.920457 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:51:43.920418    2576 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"]
May 25 11:51:43.921032 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:51:43.920783    2576 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" podUID="40b68ed6-9a1c-4ac4-929f-cbb932357de5" containerName="main" containerID="cri-o://6627ee637185c146c42e28288e7bbf36e5252340fe63ae43212010331fac60f7" gracePeriod=30
May 25 11:51:44.721629 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:51:44.721600    2576 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
May 25 11:51:44.767336 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:51:44.767213    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rvz97\" (UniqueName: \"kubernetes.io/projected/40b68ed6-9a1c-4ac4-929f-cbb932357de5-kube-api-access-rvz97\") pod \"40b68ed6-9a1c-4ac4-929f-cbb932357de5\" (UID: \"40b68ed6-9a1c-4ac4-929f-cbb932357de5\") "
May 25 11:51:44.767336 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:51:44.767269    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40b68ed6-9a1c-4ac4-929f-cbb932357de5-tls-certs\") pod \"40b68ed6-9a1c-4ac4-929f-cbb932357de5\" (UID: \"40b68ed6-9a1c-4ac4-929f-cbb932357de5\") "
May 25 11:51:44.767572 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:51:44.767324    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40b68ed6-9a1c-4ac4-929f-cbb932357de5-tmp-dir\") pod \"40b68ed6-9a1c-4ac4-929f-cbb932357de5\" (UID: \"40b68ed6-9a1c-4ac4-929f-cbb932357de5\") "
May 25 11:51:44.767572 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:51:44.767378    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/40b68ed6-9a1c-4ac4-929f-cbb932357de5-home\") pod \"40b68ed6-9a1c-4ac4-929f-cbb932357de5\" (UID: \"40b68ed6-9a1c-4ac4-929f-cbb932357de5\") "
May 25 11:51:44.767572 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:51:44.767452    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/40b68ed6-9a1c-4ac4-929f-cbb932357de5-model-cache\") pod \"40b68ed6-9a1c-4ac4-929f-cbb932357de5\" (UID: \"40b68ed6-9a1c-4ac4-929f-cbb932357de5\") "
May 25 11:51:44.767572 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:51:44.767491    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/40b68ed6-9a1c-4ac4-929f-cbb932357de5-dshm\") pod \"40b68ed6-9a1c-4ac4-929f-cbb932357de5\" (UID: \"40b68ed6-9a1c-4ac4-929f-cbb932357de5\") "
May 25 11:51:44.767572 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:51:44.767561    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/40b68ed6-9a1c-4ac4-929f-cbb932357de5-kserve-provision-location\") pod \"40b68ed6-9a1c-4ac4-929f-cbb932357de5\" (UID: \"40b68ed6-9a1c-4ac4-929f-cbb932357de5\") "
May 25 11:51:44.767864 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:51:44.767830    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40b68ed6-9a1c-4ac4-929f-cbb932357de5-model-cache" (OuterVolumeSpecName: "model-cache") pod "40b68ed6-9a1c-4ac4-929f-cbb932357de5" (UID: "40b68ed6-9a1c-4ac4-929f-cbb932357de5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:51:44.768631 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:51:44.768604    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40b68ed6-9a1c-4ac4-929f-cbb932357de5-home" (OuterVolumeSpecName: "home") pod "40b68ed6-9a1c-4ac4-929f-cbb932357de5" (UID: "40b68ed6-9a1c-4ac4-929f-cbb932357de5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:51:44.770028 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:51:44.769988    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40b68ed6-9a1c-4ac4-929f-cbb932357de5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "40b68ed6-9a1c-4ac4-929f-cbb932357de5" (UID: "40b68ed6-9a1c-4ac4-929f-cbb932357de5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 11:51:44.770028 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:51:44.770007    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40b68ed6-9a1c-4ac4-929f-cbb932357de5-kube-api-access-rvz97" (OuterVolumeSpecName: "kube-api-access-rvz97") pod "40b68ed6-9a1c-4ac4-929f-cbb932357de5" (UID: "40b68ed6-9a1c-4ac4-929f-cbb932357de5"). InnerVolumeSpecName "kube-api-access-rvz97". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 11:51:44.770484 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:51:44.770451    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40b68ed6-9a1c-4ac4-929f-cbb932357de5-dshm" (OuterVolumeSpecName: "dshm") pod "40b68ed6-9a1c-4ac4-929f-cbb932357de5" (UID: "40b68ed6-9a1c-4ac4-929f-cbb932357de5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:51:44.789011 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:51:44.788941    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40b68ed6-9a1c-4ac4-929f-cbb932357de5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "40b68ed6-9a1c-4ac4-929f-cbb932357de5" (UID: "40b68ed6-9a1c-4ac4-929f-cbb932357de5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:51:44.829091 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:51:44.829034    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40b68ed6-9a1c-4ac4-929f-cbb932357de5-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "40b68ed6-9a1c-4ac4-929f-cbb932357de5" (UID: "40b68ed6-9a1c-4ac4-929f-cbb932357de5"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:51:44.869026 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:51:44.868990    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/40b68ed6-9a1c-4ac4-929f-cbb932357de5-home\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
May 25 11:51:44.869026 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:51:44.869023    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/40b68ed6-9a1c-4ac4-929f-cbb932357de5-model-cache\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
May 25 11:51:44.869026 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:51:44.869034    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/40b68ed6-9a1c-4ac4-929f-cbb932357de5-dshm\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
May 25 11:51:44.869255 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:51:44.869045    2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/40b68ed6-9a1c-4ac4-929f-cbb932357de5-kserve-provision-location\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
May 25 11:51:44.869255 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:51:44.869055    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rvz97\" (UniqueName: \"kubernetes.io/projected/40b68ed6-9a1c-4ac4-929f-cbb932357de5-kube-api-access-rvz97\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
May 25 11:51:44.869255 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:51:44.869064    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40b68ed6-9a1c-4ac4-929f-cbb932357de5-tls-certs\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
May 25 11:51:44.869255 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:51:44.869073    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40b68ed6-9a1c-4ac4-929f-cbb932357de5-tmp-dir\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
May 25 11:51:45.278347 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:51:45.278309    2576 generic.go:358] "Generic (PLEG): container finished" podID="40b68ed6-9a1c-4ac4-929f-cbb932357de5" containerID="6627ee637185c146c42e28288e7bbf36e5252340fe63ae43212010331fac60f7" exitCode=0
May 25 11:51:45.278801 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:51:45.278398    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" event={"ID":"40b68ed6-9a1c-4ac4-929f-cbb932357de5","Type":"ContainerDied","Data":"6627ee637185c146c42e28288e7bbf36e5252340fe63ae43212010331fac60f7"}
May 25 11:51:45.278801 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:51:45.278437    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" event={"ID":"40b68ed6-9a1c-4ac4-929f-cbb932357de5","Type":"ContainerDied","Data":"d2580a9510ca9de74d92a734e335da3bae5a3ba13cf7f2bef81975894110ac4b"}
May 25 11:51:45.278801 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:51:45.278453    2576 scope.go:117] "RemoveContainer" containerID="6627ee637185c146c42e28288e7bbf36e5252340fe63ae43212010331fac60f7"
May 25 11:51:45.278801 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:51:45.278405    2576 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
May 25 11:51:45.288561 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:51:45.288544    2576 scope.go:117] "RemoveContainer" containerID="9384d5f7cd456e9180b76e0ed2fa2ca3884686694172cbae00c25ce9b14bbc19"
May 25 11:51:45.297923 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:51:45.297894    2576 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"]
May 25 11:51:45.301404 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:51:45.301376    2576 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"]
May 25 11:51:45.353247 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:51:45.353224    2576 scope.go:117] "RemoveContainer" containerID="6627ee637185c146c42e28288e7bbf36e5252340fe63ae43212010331fac60f7"
May 25 11:51:45.353616 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:51:45.353595    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6627ee637185c146c42e28288e7bbf36e5252340fe63ae43212010331fac60f7\": container with ID starting with 6627ee637185c146c42e28288e7bbf36e5252340fe63ae43212010331fac60f7 not found: ID does not exist" containerID="6627ee637185c146c42e28288e7bbf36e5252340fe63ae43212010331fac60f7"
May 25 11:51:45.353692 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:51:45.353625    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6627ee637185c146c42e28288e7bbf36e5252340fe63ae43212010331fac60f7"} err="failed to get container status \"6627ee637185c146c42e28288e7bbf36e5252340fe63ae43212010331fac60f7\": rpc error: code = NotFound desc = could not find container \"6627ee637185c146c42e28288e7bbf36e5252340fe63ae43212010331fac60f7\": container with ID starting with 6627ee637185c146c42e28288e7bbf36e5252340fe63ae43212010331fac60f7 not found: ID does not exist"
May 25 11:51:45.353692 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:51:45.353645    2576 scope.go:117] "RemoveContainer" containerID="9384d5f7cd456e9180b76e0ed2fa2ca3884686694172cbae00c25ce9b14bbc19"
May 25 11:51:45.353907 ip-10-0-132-254 kubenswrapper[2576]: E0525 11:51:45.353888    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9384d5f7cd456e9180b76e0ed2fa2ca3884686694172cbae00c25ce9b14bbc19\": container with ID starting with 9384d5f7cd456e9180b76e0ed2fa2ca3884686694172cbae00c25ce9b14bbc19 not found: ID does not exist" containerID="9384d5f7cd456e9180b76e0ed2fa2ca3884686694172cbae00c25ce9b14bbc19"
May 25 11:51:45.353964 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:51:45.353918    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9384d5f7cd456e9180b76e0ed2fa2ca3884686694172cbae00c25ce9b14bbc19"} err="failed to get container status \"9384d5f7cd456e9180b76e0ed2fa2ca3884686694172cbae00c25ce9b14bbc19\": rpc error: code = NotFound desc = could not find container \"9384d5f7cd456e9180b76e0ed2fa2ca3884686694172cbae00c25ce9b14bbc19\": container with ID starting with 9384d5f7cd456e9180b76e0ed2fa2ca3884686694172cbae00c25ce9b14bbc19 not found: ID does not exist"
May 25 11:51:47.098652 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:51:47.098540    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="40b68ed6-9a1c-4ac4-929f-cbb932357de5" path="/var/lib/kubelet/pods/40b68ed6-9a1c-4ac4-929f-cbb932357de5/volumes"
May 25 11:51:48.908388 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:51:48.908343    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5449d77-6g8hv" podUID="1c2857a2-c79b-41c3-a27d-01258dd8fdca" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/health\": dial tcp 10.133.0.44:8000: connect: connection refused"
May 25 11:51:58.908991 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:51:58.908941    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5449d77-6g8hv" podUID="1c2857a2-c79b-41c3-a27d-01258dd8fdca" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/health\": dial tcp 10.133.0.44:8000: connect: connection refused"
May 25 11:52:08.909222 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:52:08.909167    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5449d77-6g8hv" podUID="1c2857a2-c79b-41c3-a27d-01258dd8fdca" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/health\": dial tcp 10.133.0.44:8000: connect: connection refused"
May 25 11:52:18.908475 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:52:18.908430    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5449d77-6g8hv" podUID="1c2857a2-c79b-41c3-a27d-01258dd8fdca" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/health\": dial tcp 10.133.0.44:8000: connect: connection refused"
May 25 11:52:28.908820 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:52:28.908774    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5449d77-6g8hv" podUID="1c2857a2-c79b-41c3-a27d-01258dd8fdca" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/health\": dial tcp 10.133.0.44:8000: connect: connection refused"
May 25 11:52:35.179323 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:52:35.179273    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h6b5t_12b45353-8444-41fc-a8ed-fb691764eb63/ovn-acl-logging/0.log"
May 25 11:52:35.182778 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:52:35.182752    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h6b5t_12b45353-8444-41fc-a8ed-fb691764eb63/ovn-acl-logging/0.log"
May 25 11:52:38.908936 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:52:38.908893    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5449d77-6g8hv" podUID="1c2857a2-c79b-41c3-a27d-01258dd8fdca" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/health\": dial tcp 10.133.0.44:8000: connect: connection refused"
May 25 11:52:48.908160 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:52:48.908120    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5449d77-6g8hv" podUID="1c2857a2-c79b-41c3-a27d-01258dd8fdca" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/health\": dial tcp 10.133.0.44:8000: connect: connection refused"
May 25 11:52:58.908551 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:52:58.908512    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5449d77-6g8hv" podUID="1c2857a2-c79b-41c3-a27d-01258dd8fdca" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/health\": dial tcp 10.133.0.44:8000: connect: connection refused"
May 25 11:53:08.918409 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:53:08.918376    2576 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5449d77-6g8hv"
May 25 11:53:08.926678 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:53:08.926647    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5449d77-6g8hv"
May 25 11:53:09.957205 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:53:09.957172    2576 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-5449d77-6g8hv"]
May 25 11:53:10.567485 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:53:10.567442    2576 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5449d77-6g8hv" podUID="1c2857a2-c79b-41c3-a27d-01258dd8fdca" containerName="main" containerID="cri-o://a6f6da7b97d853cf0bab1ae707ef49547d4d5bfe14e2a36652aea82eeb38a546" gracePeriod=30
May 25 11:53:40.671619 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:53:40.671598    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_stop-feature-test-kserve-5449d77-6g8hv_1c2857a2-c79b-41c3-a27d-01258dd8fdca/main/0.log"
May 25 11:53:40.672043 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:53:40.671947    2576 generic.go:358] "Generic (PLEG): container finished" podID="1c2857a2-c79b-41c3-a27d-01258dd8fdca" containerID="a6f6da7b97d853cf0bab1ae707ef49547d4d5bfe14e2a36652aea82eeb38a546" exitCode=137
May 25 11:53:40.672043 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:53:40.672010    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5449d77-6g8hv" event={"ID":"1c2857a2-c79b-41c3-a27d-01258dd8fdca","Type":"ContainerDied","Data":"a6f6da7b97d853cf0bab1ae707ef49547d4d5bfe14e2a36652aea82eeb38a546"}
May 25 11:53:40.804051 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:53:40.804026    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_stop-feature-test-kserve-5449d77-6g8hv_1c2857a2-c79b-41c3-a27d-01258dd8fdca/main/0.log"
May 25 11:53:40.804388 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:53:40.804372    2576 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5449d77-6g8hv"
May 25 11:53:40.868061 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:53:40.867997    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c2857a2-c79b-41c3-a27d-01258dd8fdca-tls-certs\") pod \"1c2857a2-c79b-41c3-a27d-01258dd8fdca\" (UID: \"1c2857a2-c79b-41c3-a27d-01258dd8fdca\") "
May 25 11:53:40.868061 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:53:40.868028    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c2857a2-c79b-41c3-a27d-01258dd8fdca-home\") pod \"1c2857a2-c79b-41c3-a27d-01258dd8fdca\" (UID: \"1c2857a2-c79b-41c3-a27d-01258dd8fdca\") "
May 25 11:53:40.868061 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:53:40.868051    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c2857a2-c79b-41c3-a27d-01258dd8fdca-tmp-dir\") pod \"1c2857a2-c79b-41c3-a27d-01258dd8fdca\" (UID: \"1c2857a2-c79b-41c3-a27d-01258dd8fdca\") "
May 25 11:53:40.868316 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:53:40.868074    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pkgj4\" (UniqueName: \"kubernetes.io/projected/1c2857a2-c79b-41c3-a27d-01258dd8fdca-kube-api-access-pkgj4\") pod \"1c2857a2-c79b-41c3-a27d-01258dd8fdca\" (UID: \"1c2857a2-c79b-41c3-a27d-01258dd8fdca\") "
May 25 11:53:40.868316 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:53:40.868107    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1c2857a2-c79b-41c3-a27d-01258dd8fdca-kserve-provision-location\") pod \"1c2857a2-c79b-41c3-a27d-01258dd8fdca\" (UID: \"1c2857a2-c79b-41c3-a27d-01258dd8fdca\") "
May 25 11:53:40.868316 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:53:40.868122    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c2857a2-c79b-41c3-a27d-01258dd8fdca-dshm\") pod \"1c2857a2-c79b-41c3-a27d-01258dd8fdca\" (UID: \"1c2857a2-c79b-41c3-a27d-01258dd8fdca\") "
May 25 11:53:40.868316 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:53:40.868147    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c2857a2-c79b-41c3-a27d-01258dd8fdca-model-cache\") pod \"1c2857a2-c79b-41c3-a27d-01258dd8fdca\" (UID: \"1c2857a2-c79b-41c3-a27d-01258dd8fdca\") "
May 25 11:53:40.868642 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:53:40.868620    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c2857a2-c79b-41c3-a27d-01258dd8fdca-model-cache" (OuterVolumeSpecName: "model-cache") pod "1c2857a2-c79b-41c3-a27d-01258dd8fdca" (UID: "1c2857a2-c79b-41c3-a27d-01258dd8fdca"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:53:40.868795 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:53:40.868771    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c2857a2-c79b-41c3-a27d-01258dd8fdca-home" (OuterVolumeSpecName: "home") pod "1c2857a2-c79b-41c3-a27d-01258dd8fdca" (UID: "1c2857a2-c79b-41c3-a27d-01258dd8fdca"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:53:40.870245 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:53:40.870222    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c2857a2-c79b-41c3-a27d-01258dd8fdca-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1c2857a2-c79b-41c3-a27d-01258dd8fdca" (UID: "1c2857a2-c79b-41c3-a27d-01258dd8fdca"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 11:53:40.870357 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:53:40.870295    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c2857a2-c79b-41c3-a27d-01258dd8fdca-dshm" (OuterVolumeSpecName: "dshm") pod "1c2857a2-c79b-41c3-a27d-01258dd8fdca" (UID: "1c2857a2-c79b-41c3-a27d-01258dd8fdca"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:53:40.870357 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:53:40.870298    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c2857a2-c79b-41c3-a27d-01258dd8fdca-kube-api-access-pkgj4" (OuterVolumeSpecName: "kube-api-access-pkgj4") pod "1c2857a2-c79b-41c3-a27d-01258dd8fdca" (UID: "1c2857a2-c79b-41c3-a27d-01258dd8fdca"). InnerVolumeSpecName "kube-api-access-pkgj4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 11:53:40.887799 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:53:40.887772    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c2857a2-c79b-41c3-a27d-01258dd8fdca-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1c2857a2-c79b-41c3-a27d-01258dd8fdca" (UID: "1c2857a2-c79b-41c3-a27d-01258dd8fdca"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:53:40.938723 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:53:40.938693    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c2857a2-c79b-41c3-a27d-01258dd8fdca-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "1c2857a2-c79b-41c3-a27d-01258dd8fdca" (UID: "1c2857a2-c79b-41c3-a27d-01258dd8fdca"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:53:40.969235 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:53:40.969199    2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1c2857a2-c79b-41c3-a27d-01258dd8fdca-kserve-provision-location\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
May 25 11:53:40.969334 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:53:40.969235    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c2857a2-c79b-41c3-a27d-01258dd8fdca-dshm\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
May 25 11:53:40.969334 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:53:40.969247    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c2857a2-c79b-41c3-a27d-01258dd8fdca-model-cache\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
May 25 11:53:40.969334 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:53:40.969257    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c2857a2-c79b-41c3-a27d-01258dd8fdca-tls-certs\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
May 25 11:53:40.969334 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:53:40.969265    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c2857a2-c79b-41c3-a27d-01258dd8fdca-home\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
May 25 11:53:40.969334 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:53:40.969272    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c2857a2-c79b-41c3-a27d-01258dd8fdca-tmp-dir\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
May 25 11:53:40.969334 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:53:40.969299    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pkgj4\" (UniqueName: \"kubernetes.io/projected/1c2857a2-c79b-41c3-a27d-01258dd8fdca-kube-api-access-pkgj4\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
May 25 11:53:41.679038 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:53:41.679008    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_stop-feature-test-kserve-5449d77-6g8hv_1c2857a2-c79b-41c3-a27d-01258dd8fdca/main/0.log"
May 25 11:53:41.679487 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:53:41.679401    2576 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5449d77-6g8hv"
May 25 11:53:41.679487 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:53:41.679407    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5449d77-6g8hv" event={"ID":"1c2857a2-c79b-41c3-a27d-01258dd8fdca","Type":"ContainerDied","Data":"4051442f2423ef5c3446a9df098d1909be289467624b68b25d5d999cba15bf53"}
May 25 11:53:41.679487 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:53:41.679457    2576 scope.go:117] "RemoveContainer" containerID="a6f6da7b97d853cf0bab1ae707ef49547d4d5bfe14e2a36652aea82eeb38a546"
May 25 11:53:41.687694 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:53:41.687675    2576 scope.go:117] "RemoveContainer" containerID="e6ef237aad33fca793549ca9b90cb70d131ba5fff58de97ee935dbd56a8930b8"
May 25 11:53:41.716353 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:53:41.716324    2576 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-5449d77-6g8hv"]
May 25 11:53:41.720271 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:53:41.720252    2576 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-5449d77-6g8hv"]
May 25 11:53:43.098776 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:53:43.098742    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1c2857a2-c79b-41c3-a27d-01258dd8fdca" path="/var/lib/kubelet/pods/1c2857a2-c79b-41c3-a27d-01258dd8fdca/volumes"
May 25 11:54:39.833882 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:54:39.833849    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-798958bb55-9h57j_bb525512-86d4-401a-a262-add91f2948a7/discovery/0.log"
May 25 11:54:39.869465 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:54:39.869440    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6ddd66b668-q2wsn_39cfa0b7-b3f2-4927-abb6-996baf27f85f/router/0.log"
May 25 11:54:40.667408 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:54:40.667379    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-798958bb55-9h57j_bb525512-86d4-401a-a262-add91f2948a7/discovery/0.log"
May 25 11:54:40.696707 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:54:40.696685    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6ddd66b668-q2wsn_39cfa0b7-b3f2-4927-abb6-996baf27f85f/router/0.log"
May 25 11:54:41.477744 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:54:41.477711    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-844548ff4c-gg87l_fe69b762-18ab-4120-ba16-59a96c569243/manager/0.log"
May 25 11:54:41.488834 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:54:41.488807    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-6c886788f8-zjgpc_0657e802-a542-4e6c-b86b-22c168006885/kuadrant-console-plugin/0.log"
May 25 11:54:46.729594 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:54:46.729567    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-bx7s6_dc85864b-c44e-4ff8-b9c5-04b7ab66aeb1/global-pull-secret-syncer/0.log"
May 25 11:54:46.762529 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:54:46.762502    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-4vc6k_deccfe91-b8ff-47cc-92b2-d6f19d19ebec/konnectivity-agent/0.log"
May 25 11:54:46.854012 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:54:46.853987    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-132-254.ec2.internal_11df30495fdea7c5dad53acc2aaa0a94/haproxy/0.log"
May 25 11:54:51.268871 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:54:51.268842    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-844548ff4c-gg87l_fe69b762-18ab-4120-ba16-59a96c569243/manager/0.log"
May 25 11:54:51.294600 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:54:51.294577    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-6c886788f8-zjgpc_0657e802-a542-4e6c-b86b-22c168006885/kuadrant-console-plugin/0.log"
May 25 11:54:52.317872 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:54:52.317841    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a8391b8d-7b7f-4387-864d-63297d47d26e/alertmanager/0.log"
May 25 11:54:52.340832 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:54:52.340806    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a8391b8d-7b7f-4387-864d-63297d47d26e/config-reloader/0.log"
May 25 11:54:52.363136 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:54:52.363117    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a8391b8d-7b7f-4387-864d-63297d47d26e/kube-rbac-proxy-web/0.log"
May 25 11:54:52.384446 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:54:52.384430    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a8391b8d-7b7f-4387-864d-63297d47d26e/kube-rbac-proxy/0.log"
May 25 11:54:52.409772 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:54:52.409754    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a8391b8d-7b7f-4387-864d-63297d47d26e/kube-rbac-proxy-metric/0.log"
May 25 11:54:52.435244 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:54:52.435225    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a8391b8d-7b7f-4387-864d-63297d47d26e/prom-label-proxy/0.log"
May 25 11:54:52.457567 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:54:52.457552    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a8391b8d-7b7f-4387-864d-63297d47d26e/init-config-reloader/0.log"
May 25 11:54:52.604232 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:54:52.604216    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-84cc7cb899-fs5c9_c39eb3b0-777d-4f3c-b80f-a0b1607419d9/metrics-server/0.log"
May 25 11:54:52.634528 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:54:52.634508    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-665c9c8584-mjncd_c3f2ca6f-4b0a-47ae-ae8f-c3fce46b3e8c/monitoring-plugin/0.log"
May 25 11:54:52.668431 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:54:52.668411    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-prk6s_bd56bbf7-e0c6-420b-beea-e53649d3ff13/node-exporter/0.log"
May 25 11:54:52.702408 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:54:52.702389    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-prk6s_bd56bbf7-e0c6-420b-beea-e53649d3ff13/kube-rbac-proxy/0.log"
May 25 11:54:52.729917 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:54:52.729896    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-prk6s_bd56bbf7-e0c6-420b-beea-e53649d3ff13/init-textfile/0.log"
May 25 11:54:52.923816 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:54:52.923798    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-9c959559f-strz4_5db1ca72-24e9-4109-a52d-3e05b9682a00/kube-rbac-proxy-main/0.log"
May 25 11:54:52.945120 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:54:52.945097    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-9c959559f-strz4_5db1ca72-24e9-4109-a52d-3e05b9682a00/kube-rbac-proxy-self/0.log"
May 25 11:54:52.965948 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:54:52.965931    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-9c959559f-strz4_5db1ca72-24e9-4109-a52d-3e05b9682a00/openshift-state-metrics/0.log"
May 25 11:54:53.030967 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:54:53.030950    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_bb1f4e62-0113-40c8-b837-8f0a75d4f2c9/prometheus/0.log"
May 25 11:54:53.049760 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:54:53.049725    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_bb1f4e62-0113-40c8-b837-8f0a75d4f2c9/config-reloader/0.log"
May 25 11:54:53.073554 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:54:53.073537    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_bb1f4e62-0113-40c8-b837-8f0a75d4f2c9/thanos-sidecar/0.log"
May 25 11:54:53.097096 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:54:53.097077    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_bb1f4e62-0113-40c8-b837-8f0a75d4f2c9/kube-rbac-proxy-web/0.log"
May 25 11:54:53.119001 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:54:53.118973    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_bb1f4e62-0113-40c8-b837-8f0a75d4f2c9/kube-rbac-proxy/0.log"
May 25 11:54:53.145314 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:54:53.145291    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_bb1f4e62-0113-40c8-b837-8f0a75d4f2c9/kube-rbac-proxy-thanos/0.log"
May 25 11:54:53.166881 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:54:53.166866    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_bb1f4e62-0113-40c8-b837-8f0a75d4f2c9/init-config-reloader/0.log"
May 25 11:54:53.350111 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:54:53.350078    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-65f4778c88-nlwjj_6c5db401-082d-48c0-8393-56392d8afbcd/thanos-query/0.log"
May 25 11:54:53.376197 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:54:53.376172    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-65f4778c88-nlwjj_6c5db401-082d-48c0-8393-56392d8afbcd/kube-rbac-proxy-web/0.log"
May 25 11:54:53.403157 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:54:53.403137    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-65f4778c88-nlwjj_6c5db401-082d-48c0-8393-56392d8afbcd/kube-rbac-proxy/0.log"
May 25 11:54:53.427957 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:54:53.427939    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-65f4778c88-nlwjj_6c5db401-082d-48c0-8393-56392d8afbcd/prom-label-proxy/0.log"
May 25 11:54:53.451871 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:54:53.451853    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-65f4778c88-nlwjj_6c5db401-082d-48c0-8393-56392d8afbcd/kube-rbac-proxy-rules/0.log"
May 25 11:54:53.477079 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:54:53.477058    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-65f4778c88-nlwjj_6c5db401-082d-48c0-8393-56392d8afbcd/kube-rbac-proxy-metrics/0.log"
May 25 11:54:55.440197 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:54:55.440165    2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-must-gather-qcmnf/perf-node-gather-daemonset-qkqtf"]
May 25 11:54:55.440600 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:54:55.440578    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c2857a2-c79b-41c3-a27d-01258dd8fdca" containerName="main"
May 25 11:54:55.440644 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:54:55.440604    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c2857a2-c79b-41c3-a27d-01258dd8fdca" containerName="main"
May 25 11:54:55.440644 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:54:55.440615    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aacdbcbe-b12e-4969-9cfe-586811191878" containerName="main"
May 25 11:54:55.440644 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:54:55.440620    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="aacdbcbe-b12e-4969-9cfe-586811191878" containerName="main"
May 25 11:54:55.440644 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:54:55.440634    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aacdbcbe-b12e-4969-9cfe-586811191878" containerName="storage-initializer"
May 25 11:54:55.440644 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:54:55.440640    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="aacdbcbe-b12e-4969-9cfe-586811191878" containerName="storage-initializer"
May 25 11:54:55.440795 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:54:55.440646    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c2857a2-c79b-41c3-a27d-01258dd8fdca" containerName="storage-initializer"
May 25 11:54:55.440795 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:54:55.440651    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c2857a2-c79b-41c3-a27d-01258dd8fdca" containerName="storage-initializer"
May 25 11:54:55.440795 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:54:55.440657    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="40b68ed6-9a1c-4ac4-929f-cbb932357de5" containerName="main"
May 25 11:54:55.440795 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:54:55.440662    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="40b68ed6-9a1c-4ac4-929f-cbb932357de5" containerName="main"
May 25 11:54:55.440795 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:54:55.440678    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="40b68ed6-9a1c-4ac4-929f-cbb932357de5" containerName="storage-initializer"
May 25 11:54:55.440795 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:54:55.440683    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="40b68ed6-9a1c-4ac4-929f-cbb932357de5" containerName="storage-initializer"
May 25 11:54:55.440795 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:54:55.440740    2576 memory_manager.go:356] "RemoveStaleState removing state" podUID="40b68ed6-9a1c-4ac4-929f-cbb932357de5" containerName="main"
May 25 11:54:55.440795 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:54:55.440748    2576 memory_manager.go:356] "RemoveStaleState removing state" podUID="1c2857a2-c79b-41c3-a27d-01258dd8fdca" containerName="main"
May 25 11:54:55.440795 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:54:55.440756    2576 memory_manager.go:356] "RemoveStaleState removing state" podUID="aacdbcbe-b12e-4969-9cfe-586811191878" containerName="main"
May 25 11:54:55.445092 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:54:55.445072    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-qcmnf/perf-node-gather-daemonset-qkqtf"
May 25 11:54:55.447664 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:54:55.447643    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-qcmnf\"/\"kube-root-ca.crt\""
May 25 11:54:55.447748 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:54:55.447678    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-qcmnf\"/\"openshift-service-ca.crt\""
May 25 11:54:55.448448 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:54:55.448435    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-qcmnf\"/\"default-dockercfg-22mcc\""
May 25 11:54:55.452097 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:54:55.452078    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-qcmnf/perf-node-gather-daemonset-qkqtf"]
May 25 11:54:55.551426 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:54:55.551395    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5ae4e78d-0543-4d07-9194-ee5890f4ec6c-sys\") pod \"perf-node-gather-daemonset-qkqtf\" (UID: \"5ae4e78d-0543-4d07-9194-ee5890f4ec6c\") " pod="openshift-must-gather-qcmnf/perf-node-gather-daemonset-qkqtf"
May 25 11:54:55.551533 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:54:55.551432    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7c6p\" (UniqueName: \"kubernetes.io/projected/5ae4e78d-0543-4d07-9194-ee5890f4ec6c-kube-api-access-c7c6p\") pod \"perf-node-gather-daemonset-qkqtf\" (UID: \"5ae4e78d-0543-4d07-9194-ee5890f4ec6c\") " pod="openshift-must-gather-qcmnf/perf-node-gather-daemonset-qkqtf"
May 25 11:54:55.551533 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:54:55.551463    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5ae4e78d-0543-4d07-9194-ee5890f4ec6c-lib-modules\") pod \"perf-node-gather-daemonset-qkqtf\" (UID: \"5ae4e78d-0543-4d07-9194-ee5890f4ec6c\") " pod="openshift-must-gather-qcmnf/perf-node-gather-daemonset-qkqtf"
May 25 11:54:55.551635 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:54:55.551527    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5ae4e78d-0543-4d07-9194-ee5890f4ec6c-proc\") pod \"perf-node-gather-daemonset-qkqtf\" (UID: \"5ae4e78d-0543-4d07-9194-ee5890f4ec6c\") " pod="openshift-must-gather-qcmnf/perf-node-gather-daemonset-qkqtf"
May 25 11:54:55.551635 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:54:55.551566    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5ae4e78d-0543-4d07-9194-ee5890f4ec6c-podres\") pod \"perf-node-gather-daemonset-qkqtf\" (UID: \"5ae4e78d-0543-4d07-9194-ee5890f4ec6c\") " pod="openshift-must-gather-qcmnf/perf-node-gather-daemonset-qkqtf"
May 25 11:54:55.652598 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:54:55.652574    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5ae4e78d-0543-4d07-9194-ee5890f4ec6c-sys\") pod \"perf-node-gather-daemonset-qkqtf\" (UID: \"5ae4e78d-0543-4d07-9194-ee5890f4ec6c\") " pod="openshift-must-gather-qcmnf/perf-node-gather-daemonset-qkqtf"
May 25 11:54:55.652724 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:54:55.652606    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-c7c6p\" (UniqueName: \"kubernetes.io/projected/5ae4e78d-0543-4d07-9194-ee5890f4ec6c-kube-api-access-c7c6p\") pod \"perf-node-gather-daemonset-qkqtf\" (UID: \"5ae4e78d-0543-4d07-9194-ee5890f4ec6c\") " pod="openshift-must-gather-qcmnf/perf-node-gather-daemonset-qkqtf"
May 25 11:54:55.652724 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:54:55.652628    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5ae4e78d-0543-4d07-9194-ee5890f4ec6c-lib-modules\") pod \"perf-node-gather-daemonset-qkqtf\" (UID: \"5ae4e78d-0543-4d07-9194-ee5890f4ec6c\") " pod="openshift-must-gather-qcmnf/perf-node-gather-daemonset-qkqtf"
May 25 11:54:55.652724 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:54:55.652649    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5ae4e78d-0543-4d07-9194-ee5890f4ec6c-proc\") pod \"perf-node-gather-daemonset-qkqtf\" (UID: \"5ae4e78d-0543-4d07-9194-ee5890f4ec6c\") " pod="openshift-must-gather-qcmnf/perf-node-gather-daemonset-qkqtf"
May 25 11:54:55.652724 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:54:55.652667    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5ae4e78d-0543-4d07-9194-ee5890f4ec6c-podres\") pod \"perf-node-gather-daemonset-qkqtf\" (UID: \"5ae4e78d-0543-4d07-9194-ee5890f4ec6c\") " pod="openshift-must-gather-qcmnf/perf-node-gather-daemonset-qkqtf"
May 25 11:54:55.652724 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:54:55.652695    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5ae4e78d-0543-4d07-9194-ee5890f4ec6c-sys\") pod \"perf-node-gather-daemonset-qkqtf\" (UID: \"5ae4e78d-0543-4d07-9194-ee5890f4ec6c\") " pod="openshift-must-gather-qcmnf/perf-node-gather-daemonset-qkqtf"
May 25 11:54:55.652899 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:54:55.652726    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5ae4e78d-0543-4d07-9194-ee5890f4ec6c-proc\") pod \"perf-node-gather-daemonset-qkqtf\" (UID: \"5ae4e78d-0543-4d07-9194-ee5890f4ec6c\") " pod="openshift-must-gather-qcmnf/perf-node-gather-daemonset-qkqtf"
May 25 11:54:55.652899 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:54:55.652759    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5ae4e78d-0543-4d07-9194-ee5890f4ec6c-lib-modules\") pod \"perf-node-gather-daemonset-qkqtf\" (UID: \"5ae4e78d-0543-4d07-9194-ee5890f4ec6c\") " pod="openshift-must-gather-qcmnf/perf-node-gather-daemonset-qkqtf"
May 25 11:54:55.652899 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:54:55.652794    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5ae4e78d-0543-4d07-9194-ee5890f4ec6c-podres\") pod \"perf-node-gather-daemonset-qkqtf\" (UID: \"5ae4e78d-0543-4d07-9194-ee5890f4ec6c\") " pod="openshift-must-gather-qcmnf/perf-node-gather-daemonset-qkqtf"
May 25 11:54:55.660542 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:54:55.660522    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7c6p\" (UniqueName: \"kubernetes.io/projected/5ae4e78d-0543-4d07-9194-ee5890f4ec6c-kube-api-access-c7c6p\") pod \"perf-node-gather-daemonset-qkqtf\" (UID: \"5ae4e78d-0543-4d07-9194-ee5890f4ec6c\") " pod="openshift-must-gather-qcmnf/perf-node-gather-daemonset-qkqtf"
May 25 11:54:55.755854 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:54:55.755799    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-qcmnf/perf-node-gather-daemonset-qkqtf"
May 25 11:54:55.882648 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:54:55.877209    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-qcmnf/perf-node-gather-daemonset-qkqtf"]
May 25 11:54:55.884204 ip-10-0-132-254 kubenswrapper[2576]: W0525 11:54:55.884175    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod5ae4e78d_0543_4d07_9194_ee5890f4ec6c.slice/crio-45e997889b05ca810dc06060d68b01665ab27a9bd4b0bd74a45bfbdeac8cb5e9 WatchSource:0}: Error finding container 45e997889b05ca810dc06060d68b01665ab27a9bd4b0bd74a45bfbdeac8cb5e9: Status 404 returned error can't find the container with id 45e997889b05ca810dc06060d68b01665ab27a9bd4b0bd74a45bfbdeac8cb5e9
May 25 11:54:55.885855 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:54:55.885836    2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 25 11:54:55.921993 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:54:55.921965    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-qcmnf/perf-node-gather-daemonset-qkqtf" event={"ID":"5ae4e78d-0543-4d07-9194-ee5890f4ec6c","Type":"ContainerStarted","Data":"45e997889b05ca810dc06060d68b01665ab27a9bd4b0bd74a45bfbdeac8cb5e9"}
May 25 11:54:56.756545 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:54:56.756519    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-8lxgp_62aaed5e-f12a-4833-acbc-4ea8bfe20fb1/dns/0.log"
May 25 11:54:56.783980 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:54:56.783952    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-8lxgp_62aaed5e-f12a-4833-acbc-4ea8bfe20fb1/kube-rbac-proxy/0.log"
May 25 11:54:56.926899 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:54:56.926871    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-qcmnf/perf-node-gather-daemonset-qkqtf" event={"ID":"5ae4e78d-0543-4d07-9194-ee5890f4ec6c","Type":"ContainerStarted","Data":"b49a682bf2ee4bbab5139a521e73e6afa47cccfa57c081dd9efbfc6add2ba120"}
May 25 11:54:56.927063 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:54:56.926957    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-qcmnf/perf-node-gather-daemonset-qkqtf"
May 25 11:54:56.942933 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:54:56.942894    2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-must-gather-qcmnf/perf-node-gather-daemonset-qkqtf" podStartSLOduration=1.9428829570000001 podStartE2EDuration="1.942882957s" podCreationTimestamp="2026-05-25 11:54:55 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 11:54:56.940955266 +0000 UTC m=+2242.458686720" watchObservedRunningTime="2026-05-25 11:54:56.942882957 +0000 UTC m=+2242.460614413"
May 25 11:54:56.953901 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:54:56.953875    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-xn288_4ab64dbc-4ab2-4199-bf57-698ec5539507/dns-node-resolver/0.log"
May 25 11:54:57.392788 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:54:57.392747    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-685f589447-dtpd9_e1e82e94-6cf9-4ca9-a606-fe51ed2aaf29/registry/0.log"
May 25 11:54:57.412578 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:54:57.412550    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-nkcf5_cd7634d4-8eb9-42a5-9e01-7d94645afbd3/node-ca/0.log"
May 25 11:54:58.406741 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:54:58.406711    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-798958bb55-9h57j_bb525512-86d4-401a-a262-add91f2948a7/discovery/0.log"
May 25 11:54:58.456672 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:54:58.456650    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6ddd66b668-q2wsn_39cfa0b7-b3f2-4927-abb6-996baf27f85f/router/0.log"
May 25 11:54:58.896900 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:54:58.896875    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-cpqmb_c25259af-1ae3-465a-a802-2dd6fe637e06/serve-healthcheck-canary/0.log"
May 25 11:54:59.396222 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:54:59.396181    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-8kl5j_3d0e9f3f-1d89-4a43-b947-a242f4cf2e52/kube-rbac-proxy/0.log"
May 25 11:54:59.417908 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:54:59.417883    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-8kl5j_3d0e9f3f-1d89-4a43-b947-a242f4cf2e52/exporter/0.log"
May 25 11:54:59.439350 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:54:59.439300    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-8kl5j_3d0e9f3f-1d89-4a43-b947-a242f4cf2e52/extractor/0.log"
May 25 11:55:02.124204 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:55:02.124176    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-757cd88c59-mqdwf_1315774b-398d-4228-81a3-1e9d0001ccc8/manager/0.log"
May 25 11:55:02.939471 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:55:02.939441    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-qcmnf/perf-node-gather-daemonset-qkqtf"
May 25 11:55:09.710788 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:55:09.710757    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-dgspq_ac23d02e-4fdd-4cfe-a16a-0958e2a7b62e/kube-multus-additional-cni-plugins/0.log"
May 25 11:55:09.734475 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:55:09.734449    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-dgspq_ac23d02e-4fdd-4cfe-a16a-0958e2a7b62e/egress-router-binary-copy/0.log"
May 25 11:55:09.761081 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:55:09.761057    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-dgspq_ac23d02e-4fdd-4cfe-a16a-0958e2a7b62e/cni-plugins/0.log"
May 25 11:55:09.782784 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:55:09.782759    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-dgspq_ac23d02e-4fdd-4cfe-a16a-0958e2a7b62e/bond-cni-plugin/0.log"
May 25 11:55:09.804698 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:55:09.804677    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-dgspq_ac23d02e-4fdd-4cfe-a16a-0958e2a7b62e/routeoverride-cni/0.log"
May 25 11:55:09.827595 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:55:09.827579    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-dgspq_ac23d02e-4fdd-4cfe-a16a-0958e2a7b62e/whereabouts-cni-bincopy/0.log"
May 25 11:55:09.855232 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:55:09.855214    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-dgspq_ac23d02e-4fdd-4cfe-a16a-0958e2a7b62e/whereabouts-cni/0.log"
May 25 11:55:09.963291 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:55:09.963204    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-qjgcp_a2602414-6828-4ead-beff-a1e1215f9851/kube-multus/0.log"
May 25 11:55:09.985688 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:55:09.985661    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-5859j_ff50c150-45a9-41c6-be52-886f2c960e85/network-metrics-daemon/0.log"
May 25 11:55:10.005205 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:55:10.005182    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-5859j_ff50c150-45a9-41c6-be52-886f2c960e85/kube-rbac-proxy/0.log"
May 25 11:55:11.350082 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:55:11.350050    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h6b5t_12b45353-8444-41fc-a8ed-fb691764eb63/ovn-controller/0.log"
May 25 11:55:11.367692 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:55:11.367665    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h6b5t_12b45353-8444-41fc-a8ed-fb691764eb63/ovn-acl-logging/0.log"
May 25 11:55:11.379682 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:55:11.379658    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h6b5t_12b45353-8444-41fc-a8ed-fb691764eb63/ovn-acl-logging/1.log"
May 25 11:55:11.399162 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:55:11.399135    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h6b5t_12b45353-8444-41fc-a8ed-fb691764eb63/kube-rbac-proxy-node/0.log"
May 25 11:55:11.426320 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:55:11.426287    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h6b5t_12b45353-8444-41fc-a8ed-fb691764eb63/kube-rbac-proxy-ovn-metrics/0.log"
May 25 11:55:11.445620 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:55:11.445602    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h6b5t_12b45353-8444-41fc-a8ed-fb691764eb63/northd/0.log"
May 25 11:55:11.466263 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:55:11.466245    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h6b5t_12b45353-8444-41fc-a8ed-fb691764eb63/nbdb/0.log"
May 25 11:55:11.487360 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:55:11.487340    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h6b5t_12b45353-8444-41fc-a8ed-fb691764eb63/sbdb/0.log"
May 25 11:55:11.585429 ip-10-0-132-254 kubenswrapper[2576]: I0525 11:55:11.585409    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h6b5t_12b45353-8444-41fc-a8ed-fb691764eb63/ovnkube-controller/0.log"